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Introducrion and Overview 



Michael Nugent and James L. Ratcliff 

The first international working conference on ^'Measurements of Quality in Higher 
Education- was held May 23-24, 1991, in Washington, D.C.. The conference was 
sponsored by the Center for the Study of Higher Education (CSHE) and the National Center 
for Teaching Learning and Assessment (NCTLA), the Pennsylvania State University. 
Conference participants met to examine the possibility of establishing an interinstitutional 
cooperative research agenda that focused on factors relating to the cross-national 
measurement of quality in higher education. 

This volume contains papers that outline fundamental concepts and research interests 
connected to the measurement of quality in postsecondaiy education. Participants reported 
on research on measures of quality that was at different stages in their respective countries. 
Some were beginning to formulate research projects, while others were associated, with on- 
going research endeavon. As a result, some of the presentations consisted of concept papers 
while others were more extensive reports of research underway or findings from recently 
concluded studies. In that sense, this collection of papers reflects the int^national variation 
in the course and status of such research. 

Purpose of the Mav Meeting 

The sponsors of the May meeting set out to c^italize on the interest in educational 
quality stimulated by the establishment of National Educational Goals in the United States. 
These goals were outlined in America 2000 . a repon sponsored by President Bush. The 
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relationship \ these goals and quality in U.S. higher education is addressed by Sal 
Conallo in .y in Higher Education and the National Education Goals." Corrallo 
discusses tiie i. ors involved in assessing college graduates* ability to think critically and 
solve problems. Similarly, Michael Netties* paper, "The Role of American Higher 
Education in Making America First in tiie World in Science and Matiiematics by tiie Year 
2000," examines the role of quality assessment in tiic context of tiie objectives put fortii in 
America 2000 . Although Netties focuses specifically on matiiematics and science, he raises a 
fundamental issue: If tiie United States, or any otiier nation, intends to establish a high- 
quality educational program relative to international standards, it must develop a systematic 
understanding of otiier countries' metiiods of measuring and assessing quality. 

At the May meeting, four specific research ideas were established as areas of interest 
for international collaborative research. 

1. Litemational assessment of student learning in specific disciplines. 

2. Hie impact quality assessment on higher education systems. 

3. Drop-out, retention, and enroUment pattons of students. 

4. Hie role of peer review in the evahiation/assessment process. 



The September Meeting 

Following tiie May 24 meeting in Washington, D.C., tiie Center for ffigher Education 
Policy Studies (CHEPS), " 'ersity of Twente, tiie Netiierlands, and tiie Center for tiie 
Study of Higher Education, 1. ivania State University, sponsored a second 

international meeting on tiie measure, iit of quality in higher education. The meeting was 
held at tiie Napier Polytechnic, Edinburgh, S4X>tiand, on September 5, 1991, immediately 
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after the European Association of Institutional Research conference (EAIR). The September 
meeting was attended by researchers from Belgium, Canada, Finland, Germany, Hong Kong, 
the Netherlands, Sweden, the United Kingdom, and the United StaUJs. The purpose of the 
second meeting v^s to continue the work started at the first meeting; the research agendas 
established during the Washington meeting were outlined and decisions were made about 
countries and individuals that would become research partners. Each of the four research 
ideas were examined according to the following points: 

1. Literest of various parties in particular research endeavors (not all 
investigators will be interested in pursuing research in all four areas. 
Certain researchers may only be interested in undertaking research on 
one specific topic.) 

2. Level of participation (some researchers may want to participate as 
key investigators on some projects, while others may want to play an 
advisory role.) 

3. Scope of research Qnvestigation on many levels, including 
international, systemic, federal, state, or institutional.) 

4. Coordination of research (those researchers who will coordinate the 
investigation, eitho' on a collaix)rative or individual basis.) 

5. Investigative models 

6. Research methods 

7. Sources of funding 

At the outset it was assumed that, given the number of participants at the Edinburgh 
meeting, the interest of group members would vary relative to the project discussed. After 
project interests were ascertained, several levels of interest were proposed. The cosponsors, 
CSHE, NCTLA, or CHEPS, did not intend to act the principal coordinators of any research 
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activities disc at this meeting, although all are engaged in research activities related to 
one or more oi proposed issues. Sponsors did not intend to control or coordinate the 
development of research proposals. 

At least three ^roaches can be used to outline participation: 

1. Project coordinators and primary researchers 

' Individuals who have strong interests in one or more of the proposals 
may take on much more of the responsibility for funding and 
implementation. 

2. Active researchers 

Individuals would participate in design and implementation within their 
home environments, and would provide a concomitant portion of the 
funding needed to cany out these activities. 

3. Participants 

A third possible sole was site participant in the research; instead of 
accepting the respmsibilities for conducting research, an individual may 
be a site or national liaison, setting up interchanges with people actually 
involved in that research. Site participants would help identify the 
individuals and institutions in their home country tiiat were interested in 
collaborative research in this area. 



This Volume 

Whereas the complexity of defining educational quality in a single national system is 
significant, the complexity of determining quality at the international level is magnified by a 
many additional interacting cultural and societal factors. Quality as defined within different 
societies, higher education systems, and institutions, and among faculty members and 
students, poses interesting problems indeed. Jeffery Gilmore's paper, 'Concqptualization of 
College Quality," underlines this pjint by describing a number of different concq>tions of 
quality in U.S. undergraduate education. He cauticms, though, that a concq)tual framework 

6 



8 



for deteraiining quality on an international level should consider cross-national 
distinctiveness. In his psqwr, ' New Approaches to Evaluation in Austrian Higher 
Education," Hans Pechar points out that although centrsdizing mechanisms within the 
Austrian higher education system do not emphasize quaUty performance at the institutional 
level, an increasing amount of attention has nevertheless been paid to the role of individual 
institutions as mechanisms for increasing quality. In this instance, the structural 
characteristics of weak institutional autonomy are overlooked in attempts to emphasize 
quality assurance at the individual institutions. At the other end of the spectrum is the 
British higher education system. Unlike Austrian higher education, British higher education 
is founded on a much greater degree of institutional autonomy and identity. John Brennan 
outlines the various levels and dimensions of quality assessment in "Questions of Quality in 
UK Higher Education." In addition, he discusses the potential impact of new policies 
proposed in the White Ps^ published by the British Government in May 1991, as well as 
two on-going research projects on the assessment of quality. 

Several conceptual framewoiks and taxonomies have been presented in this volume to 
categorize mechanisms involved in quality assurance. David Woodhouse provides one 
description in "Taxonomy and Comparison of Approaches to Quality Assurance in Higher 
Education." His taxonomy is separated into type of body (government, institution, etc.), 
scope (institution, discipline, etc.) and authority within the institution (external, internal, 
etc.). Edgar Frackmann describes a framework for understanding quality assurance cross 
nationally in "Comparative Aspects of Quality Assurance in Higher Education." IBs 
framework is presented in the form of a matrix, distinguishing between internal/external and 
implicit/explicit quality assurance mechanisms in which fectors such as self-regulation, peer 
review, market mechanisms, and government control manifest themselves. 

The difficulties of understanding assessment across national higher education systems 
is discussed by Trudy Banta and Homer Fisher in "An International Perspective on Assessing 
Baccalaureate Program Outcomes." Though differences in higher education systems and 
assessment procedures exist not only among European countries but also across the Atlantic, 
there are also important similarities. Academic institutions struggle to retain autonomy in the 
face of increasing demand for external control. One main roadblock to international 
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compa s been the lack of comprehensive cross-national data on higher education 
system. tudents based oa an international comparative framework. E. Stqphen Hunt 
addresst s issue in "Developing Tools for Using National and International Data Bases. " 

Ol issues presented in this volume deal with the importance of research, student 
persistence, and philosophical and ideological £v:tors in defining quality assurance. In "The 
Research-oriented Dynamism of Education," Jose Strubbe discusses the role of research in 
detraining educational quality. According to Strubbe, dynamic research plays an important 
role in establishing programmatic, curricular, and institutional quality in Belgian universities. 
He believes that research policy is an important determinant of educaticmal quality. The 
relationship between student persistence and quality is discussed in two ^apea. In "Using 
Student Persistence Research to Strengthen the Quality of fbt Teaching and Institutional 
Process," Robert Froh advocates the use of student persistence as a factor in the analysis of 
quality cross natioa?: discusses Riekele Bijleveld*s pi^, "Eiqilaining Differences 

in Drop-out Between i'acuities in Dutch University Educatimi" (also included in tiiis volume) 
in which Bijleveld compares drop-out levels between different Dutch faculties, or 
departments. Bijleveld points out that a study of the substantial differences across Acuities 
in the Netiierlands in student drop-out rates promises to provide an informative cmitrast 
between traditional and innovative programs. Specifically, ffijlevdd examines an innovative 
program at the University of limberg in contrast to the more traditional Dutch university. 

The philosophical and ideological nature of quality assessment is touched upon in the 
conclusion of "Quality Assessment in Postsecondary Education: A Mexican Perspective" by 
Felipe Rizo. Rizo points out the importance of recognizing that questions related to the 
evaluation of effectiveness aad efficiency are fundamentally philosophical and ideological in 
Mexico. Viewed in light of international comparisons, the philosophical and ideological 
nature of these questions multiply, becoming a challenge to comparative researchers. 

This volume acts as tiie starting point fiir future cooperative cross-national research in 
higher education. ^^y meeting, a second meeting was held on September 5, 1991, 

in Edinburgh, Scotiand, whica was co-sponsored by CSHE and CHEPS. A third meeting, 
sponsored joinUy by die Council for National Academic Awards (CNAA), CSHE, and 
CHEPS has been planned for April 15 and 16, 1992, in London, England. These jointiy- 




sponsored meetings are evidence of the successful establishment of cross-national, 
interinstitutional relationships. Participants are listed in the ^>pendix for readers who wish to 
contact authors of particular papers or to contact researchers from participating countries for 
additional information. 



Quality in Higher Educarinn ai^^f 
the National Education finals 

Sal Conallo* 
U.S. Department of Education 



Background 

HistoricaUy, the identification and measurement of quaHty in higher education has 
T^r ^ Everyone talks about it but few can do anything about 

It. In 1989 the National Center for Postsecondary Governance and Finance conducted a 

H^^l^^SS""* K V^"^'^ " ^1 Academy.- At the symposium. Estela Bensimon 
noted that. Although there is renewed interest in quaUty among key actors in leadership 
positions, little IS known about what they~and various institutional and systemic actors-raean 
when they call for quaUty." Four perspectives on quaUty in the academy were expressed in 
I»pers pre^ted at the symposium. The views included those of the trustees, the president 
planmng officers, and state poHcy makers. presiacni, 

th. ,„»h«r?^ discussion of Frank Schmidttein's plamiing-focused paper, Bensimon reports 
the author s suggestion that "... just about everything an institution does and is can be 
considwed withm the context of quaUty." In Robert Bimbaum's piece, he offers this advice: 
... the ^anous dimensions of quaUty often have structural or procedural requirements that 
arc m conflict. Bimhaam sugg^ts that fixed tradeoffs exist among teaching, services and 
research (some might call if an '•Tron Law of Faculty OuQiuts"), so that devoting time to one 
may mc^ that the others w/l| suffer. In the section of the Bensimon paper entiUed "Issues 
to Considw, the rcader is urged to keep three issues in mind whUe rcviev4ng the paper 
TTic second issue is measuring quaUty on crmpus. She asks, "Does it matter that inStational 
leaden tend notto define quaUty in terms of the development of the total student, in both the 
cogmtive and affective donudns?"* 



'TWs paper was prepared by the author in Ms private TTie views expressed are 

Ae author s. No official support by the U.S. Department of Education is intended or should 

DC infCITCQ. 

^tda M. Bensimon, "QuaUty in the Academy: Proceedings from a National 
Symposium," Nationat Center for Postsecondary Governance and Finance Universitv of 
Maryland, CoUege Park, Maryland, 1989. ^ 
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Although the papers do not necessarily suggest a lack of concern with student 
achievement as a measure of quality on campus, the call for just such an evaluation of 
college learning by the President and the governors suggests that others feel differently. To 
that end, National Education Goal #5 was formulated: By the year 2000, every adult 
American will be literate and possess the knowledge and skills necessary in a global economy 
and exercise the lights and re^nsibilities of citizenship. Goal #S includes an objective 
directed at the learning of higher order thinking and communication skills of college 
graduates. According to that objective, "The proportion of college graduates who 
demonstrate an advanced ^ility to think critically, communicate effectively, and solve 
problems will increase substantially." 

The U.S. Dqpartment of Education is currently worlong to identify one or more 
approaches to assessing achievement of this objective. This will not be an easy or short-term 
task. The National Goals Resource Group's March 1991 Interim Report noted that 

. . . neither national nor state information is currently 
available on the ability of college graduates to "think 
critically, communicate effectively, and solve problems. 

Further, 

If the National Goals Panel wishes to assess the ability of college 
graduates to think critically, to communicate effectivdy, and to solve 
problems, a new kind of assessment will have to be created. That assessment 
might be a type of Nntional Assessment of Educational Programs (NAEP) at 
the college level, given to a national sample of college students at different 
kinds of schools across the Nation. To have credibility, such an assessment 
would have to take into account differences in the postsecondaxy institutions in 
America and the fiact that the pluralistic system in place today has extended 
postsecondaxy educational opportunities to the broadest cross section ever of 
America's citizens. Developing a NAEP-like assessment would be 
controversial for many reasons. It would require S years or more to develop 
and an investment of several scores of millions of dollars to make 
operational.^ 



Key Issues and roncems 

In considering the development of an assessmoit strategy, four concerns or issues 



^National Educational Goals Resources Group, Interim Report, March 1991. 
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must be adc 

£ilSt. i must be a common understanding and ultimately agreement about the 
specific skills . dents are expected to have achieved in light of the larger goal of developing 
a competitive work force and assuring responsible citizenship. Not only must these skills be 
clarified, but they must be defined in a manner that allows the impact of the educational 
experience to be assessed. In addition they should be defined fiom a teaching/learning 
perspective, so tiiat tiieir enhancement can be factored into classroom experiences? 
Specifically, how can tiiese skills be enhanced by classroom teaching/learning experiences/ 

Sfi^md* how much can students be expected to know; what levels of learning or 
performance standards are to be attained? Students do not come into postsecondary 



* A 1987 report which provided suggestions for the State of New Jersey's Collie 
Outcomes Program, used much the same language when referring to tiie "broad based 
common sidlls that are necessary in all disciplines and fields.* More specifically they 
"include analysis, problem solving, critical thinking, quantitative reasoning, and written and 
oral expression. These skills are seen in a stiidents ability to find, use, and present 
information." Further elaboration follows: 

"These are skUls necessary to critically analyze and utilize information (sometimes 
referred to as "higher order" skills). Specifically tiiey include tiie skills necessary to: 

a) Accumulate and Examine Information - including the sidlls necessary to : 
determine the kinds of information needed for a given task; construct and impl«nent a 
systematic search procedure, uring both traditional and computerized methods; discard or 
retain information based on initial screening for relevance and creditability; and develop 
abstract concepts a^ropriate to tiie task at hand for initially ordering information which is 
retained. 

b) Reconfigure, Think About, and Draw Gmdusions from Information - including 
skills necessary to: evaluate the interpretations presented by others in terms of their 
assumptions, logical inferences, and empirical evidence; reconfigure information in ways that 
suggests ranges of alternative interpretations and evaluate their relative merits; cmstnict 
hypotheses that logically extend thought from areas in which information is already available 
into areas where it is ^r-- - .;dfy the additional information which might confirm or 
disconfirm those hypoui^ ' draw conclusions based on all of the above. 

c) Present Information - Including the skills necessary to express one's own ideas in 
written, oral and graphic forms which wi<^ be intelligible and persuasive to a variety of 
audiences." 
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education with the same learning experiences, family backgrounds, or cognitive abilities. 
Keeping the standards low reduces the value of the program, while making them too high can 
be troublesome and peiliaps unrealistic for students and institutions alike. Is it realistic to 
expect all to leave with the same set of skills? What level should they be expected to have 
achieved to utilize those skills? 

Third , can the attainment of these skills be measured? Are reliable assessment 
procedures available in the open market that can be used, in whole or part, to measure 
learning experiences? Can these skills, which relate more to post-school ^plication on the 
job and as a citizen, be measured that soon after the college experience? Can such tests 
maintain relevance and reliability over time? Should the assessment of student learning be 
limited to the national level? Can assessment programs, if properly designed and 
implemented at the state or institutional level, be aggregated to a national level? Lastly, 
some suggest that much of the information needed is available in current data banks. In lieu 
of collecting new information, are proxy measures available, at least for short-run use? 

Fourth , who should be tested-some or all of the students? Can the same assessment 
procedures be used for all students and all learning experiences in all institutions (i.e., for 
two-year graduates, four-year students, graduate students, and professional school graduates, 
among other groups)? How often should the tests be administered? The State of New Jersey 
has a similarly focused testing program that assesses students prior to entry into the junior 
year of college. Who will do the assessment— institutions, states, private agencies, the 
federal government? What problems arc to expected in Isurgc-scale testing? How much will 
a testing program cost and who will pay? 

Current Experiences 

In addressing these issues and concerns, there are a number of current activities and 
experiences from which valuable information can be drawn. Each of these activities provide 
unique insights into issues and concerns noted above. These include: 

1. State Assessmant Experiences : A number of states have or are planning to 
implement postsecondary assessment programs.* The State of New Jersey has been notable 
for its postsecondary assessment activities. Florida and other states have also implemented 
state-level assessment programs that can provide valuable insights for a national assessment 
process. 

2. Institutional A ?rSC.1S TP^"t Activities : A number of institutions have been in the 
forefront of the modem assessment movement, many assessing the general intellectual skills 



* rhri^tin^ P PQ..icr.n, <!fatft Tnitiatives in Assessment and Outcomes Measurement: 
Tools for Tea ching m<i T.fia P'"g 19 9Q's: Individual State Profiles. Educational 
Commission of the States, Denver, CO, May 1990. 
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of graduatir 'ents. Increasing attention to institutional assessment is coming from 
regional acc ng agencies.* 



3. ReiaLionship to Pre~Coll egiate Testiny : The assessment of the thinking and 
communication skills of college graduates is also called for under National Education Goal 
#3 for grades four, eight, and twelve. There are numerous ^roaches for the testing of 
these skills at the postsecondary education level. They include the defuiing and testing of 
basic and more advanced skills. New York' and Vermont are two examples. 

4- Testing Services College Testin g Experiences ! Both the American College Testing 
Service (ACT) and Educational Testing Services (ETS) have a number of cuiient activities 
focused on the assessment of basic learning skills at the college level. The ACT College 
Outcomes Measures Program is said to assess communication ability, problem-solving, and 
value clarification for three content areas: functioning witiiin social institutions, using 
science and technology, and using die arts. The ETS Higher Education Assessment Prognun 
has a series of major field achievement instruments as well as otiier assessment instruments 
on the shelf. In addition, a number of smaller testing service companies, such as The 
California Academic Press, have developed more narrowly focused instruments for assessing 
critical tiiinking, writing, and otiier skills. 

5. Job Skills TsCTies- American industry, we are told, must upgrade its productive 
capability to keep up wifli tiie rest of die world. This means both upgrading capital 
equipment and related work-force skills. Given die new equipment and production 
processes, industry is constantiy feced witii ascertaining tiie skills needed to perform a job in 
this new environment It should be noted tiiat die Secretary's Commission on Achieving 
Necessary Skills, U.S. Department of Labor, plans to address Uiis and odier issues through 
die U.S. Department of Education's National Adult Literacy Study cunentiy in die design 
stage. An extensive set of worlq}lace skills is identified in dieir June 1991 report, including 
diose identified in die national goal.' Examples of the application of skills are provided 
from a workplace perspective. Aldiough many carry over to die practice of citizenship, most 



• Western Association of Schools and Colleges, "Achieving Institutional Effectiveness 
Through Assessment: A Resource Manual to Support WASC Institutions," Working Paper. 
Final Draft, June 1991. 

' New York State Education Department. Basic and Expanded Basic Skills! Skills for 
Validation Study. Albany, NY, July 1990. 

' The Vennont State Department of Education, "Vermont Writing Assessment: The 
Pilot Year," MontpeUer, VT, September 1990. 

' The Secretary's Commission on Achieving Necessary Skills, "What Work Requires of 
Schools: A SCANS Report for America 2000," U.S. Department of Labor, Washington, 
D.C., June 1991. 
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are unique to the workplace. 



Bevond the Assessment of Higher Order Thinking and Communication Skills 

Goal #5 focuses on the enhancement of critical thinking, problem solving and 
communication skills. Paul R- Pintrich, in a paper prepared for a U.S. Department of 
Education-sponsored conference on postsecondary assessment, referred to an increased 
interest in critical thinking and problem solving. He suggested that 

Although this attention to the topic is interesting and exciting, there is a great deal of 
theoretical confusion concerning the nature of critical thinking and problem solving. 
Researchers and practitioners are addressing the topic from a multitude of 
perspectives and the constructs are not clearly defined. 

He cites Glaser (1984), who believed that five issues must be considered in future 
research on general cognitive skills. Pintrich implies that these questions can be thought of 
as guiding principles for the assessment of student learning. These have been par^hrased: 

1) Consideration must be given to how general cognitive skills are or can be 
taught. 

2) Consideration must be given to how current knowledge-based models of 
cognition are applied to the teaching of general problem*solving skills. 

3) Consideration must bt given to how instruction fosters general cognitive 
skills. 

4) Consideration must be given to how cognitive skills learned in one domain 
are transferred to another domain. 

5) Consideration should be given to the effectiveness of the teaching process 
for critical thinking and other general cognitive skills. 

Pintrich argues that the most effective assessment of critical thinking and problem 
solving takes place when a theory or model of how instruction will lead to criticsd thinking or 
problem solving, has been defined and is being tested. Assessment of the process would 
include information on the s^ropriateness of the curriculum and how the subject matter is 
taught, the current status and preparation of both faculty and student, the institutional and 
faculty support and commitment for the process of teaching and leaining. He concludes that. 

Regardless of the level of the local model (e.g., course dq)artmental, insrltutional), 
the development of a local model of critical thinking would help to ddint^te how Uie 
"independent" variables of course tasks and activities, curriculum offerin;;^s, and/or 
institutional dimensions theoretically influence the "dependent" variable of students' 
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He goe: that, 

from this kind of assessment program would provide very useful feedback to 
in. rs (or other program designers) about the relative efKcacy of different aspects 
of ih. f program." 

Assessment then should not merely assess the level of skills achievement, but also 
should identify the determinants that enhance the learning of these skills. An assessment 
process that only measures outcomes is limited and not consistent with the intent of Goal #5. 

Towards Quality Through Assessment 

We currently have a national weather service that provides hourly assessments of 
weather conditions across th on, even identifying the causes of those conditions. Yet 
even if we wish to, we c? .><:ue with the weather other than enjoy it when it is pleasant 
or run for shelter when . ^ dad. However, unlike the weather many things can be done to 
enhance the quality of students' educational experience. The development and 
implementation of a system to assess student learning is a necessary first stq>. It can, if 
properly designed, provide a means of comparing the performance of graduates by prcq^rams, 
schools, and states. It can also be used to determine the evenness of learning ei^Nsiences 
across course and program domains. It can isolate, firom a program perspective, effective 
and less-effective programs and practices and perhaps determine if there is an "Lx)n Law of 
Faculty Output." 

In sum, quality improvement cannot occur at the college level without an effective 
and rsliable assessment of the teaching/learning process. Unlike attempts to control the 
weather, much can be done to keep the sun shining and the student learning in the college 
classrooms of the nation. 



" Paul R. Pintrich, "Assessing Student Progress in Collie: A Process-Oriented 
Approach to Assessment of Student Learning in Postsecondaiy Settings" in Postsecondary 
Assessment Confensnce: Report of the Planning Committee. November 20, 1986, U.S. 
Department of Education, Washington, D.C. 
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The Role of Ameriran H^g ^ftr Education in Maldnp America First 
in the WnrlH in Scienc e and Mathematics bv the Year 200Q 



Michael Nettles 
University of Tennesee 

Introduction 

America's colleges and universities have a vital role to play in helping to achieve 
Goal #4, "U.S. students' becoming first in the world in mathematics and science achievement 
by the year 2000. " But many gs^s need to be filled in our knowledge and understanding 
about the present status and fimction of American mathematics and science as well as about 
the role of higher education toward advancing mathematics and science education before a 
meaningful strata can be foiged toward achieving the goal. The following questions must 
be addressed in order to fiU the knowledge void: (1) what contributions arc being made by 
America's colleges and universities toward the international conqpetitiveness of the United 
States a mathematics and science?; (2) what is the cunent gualitj: of the contributions by 
American colleges and universities in mathematics and science teaching and learning?; 
(3) how do the contributions of U.S. colleges and universities compare (amount and quality) 
with those of colleges and universities in the rest of the worid?; and (4) how can higher 
education's efforts toward making America first in the worid in mathematics and science, be 
improved? 

In this paper, methods are suggested that will assist America's coUeges and 
universities develop a strategy for understanding and improving their contributions to Goal 
#4. Methods are also suggested concerning the provision of data and information needed for 
setting specific benchmarks and monitoring progress toward achieving the goal. Since Goal 
#4 has a world rather than a national context, sttat^es for conducting international 
comparative studies, for goal setting and monitoring progress, also are proposed here. 

American higher education's contributions toward becoming first in the world in 
mathematics and science should be viewed as follows: 

1. Helping to improve mathematics and science instruction as well as outcomes in the 
primary and secondary schools in the following ways: 

A. by producing more and better prepared 
mathematics and science teachers, and 
encouraging mathematics and science majors to 
enter teaching careers; 



B« by retraining and providing continuing 
professional development for existing mathematics 
and science teachers by sharing ftcilities, 
fiaculties, technology, and summer institutes, etc.; 

C. by providing alternative routes to licensing and 
certification so that more of the talented 
mathematics and science teacher prospects will 
become interested in entering the teaching 
profession; 

D. by developing better ways of assessing 
teachers' and students' knowledge of mathematics 
and science and their skills in using mathematics 
and science in the teaching and learning process; 

E. by producing early intervention strategies for 
young people to inform them about their 
mathematics and science career options and 
encouraging them to acliieve in mathematics and 
science. 

2 StTMgthening the undergraduate mathematics and science cuiricula for 
W college students majoring in mathematics and science disciplines 
(b) students prq»ring to become teachers of mathematics and sden^ in 
TS^T^ and secondary schools, and (c) students who are not majoring in 
mathematics and ,r,^cc, but are pursuing associate and buxalaureate dLees 

Z^^''"'" ^ mathematics and science courses as part of^ 
genenJ education requirements. 

3. Integrating mathematics and science into the core or general education 
cumcula required for aU postsecondary students. 

4. Developing and improving the technology and teaching strategies so that 
students are encouraged and motivated to study and enjoy leaminE 
mathematics and science in college. 

rn,i mF'^J^ °^ these four roles, training more teachers, is expUdt in the objectives of 
f fl* • ^ *~"'*' *^' ^ "^^y important, but are ori^ imriSflv 

^^J"'^*^' Consequenay,4ny1^Ieiai^^ 
^^Sv'^Ti: to elementary and secoiS^ sch^ce S 

m the objective IS placed up^r ^y grades." However, the need is equaUy urgentTtte 
coUege level, and in fact improvements in the eariy grades are partiaUy ZiiSi^n 
improvements in higher education. F~u-uy ucpcnacni upon 

The need to strengthen college curricula in mathematics and- science may be 
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<«orch productivity of Uk U.S. as m«a«d by SISS^ fjA^TT 

by *c n^mb^A^S-ciS^^^Srs'tS^s",;?^^ '^'^ 
mathematics and science di«nnUn*^ n^r k« ♦f.^ s, master s, and doctoral degrees in 

Amencans majoring to mathenutics and science at sdSTl^S^i- »"?^.<>f 

or u.te™tional assessments of nuthenmics and Sou*^^.^^,?^"*- "1^ "f"?* 

do not peniiit U.S. associate and baccalaureate <^«Sr, K?^ 

countetpws abroad. E«minatio.7^^S^"K^ ^T™* 

nom, m U.S. colleges and umversities as they a« to IS^^'S """^ 

science^'^enlTaJ^Sr??? '•""^ 
baseline aU levels in U.S. colleges and univetsities. Iliese data i»ovide a 

baseto* (1991) tlm can be used to momtor progress towarf tweisfag tMs Sd 
Cooperative anangements need to be established wiaioUierii«tin».:.«_. . 

d^s no, detStSTX^^Sc^r^S^ ««N..e evide.^ 

^r^I^^ZTT^l An^icanlSS^' 
i^Z^u^^^^"^ nuyor-have acquired the mathematics and scimm aIiu 
■taough . genetal eduction cuniculum flutt will male ttaTSSSfy 

Hw 6a tint American colleges and univetsities attemnt >« t. 
than a select few as to most other Sions, saggcS^S^SL^hSl"^ 
««de.p»d.ate matitematics and science ducJK^ „ t^^^^ " 

S ?^ ~ ™« » for primaty tS^Sto d,e 



Monitoring Prffgrrff! 



How will the nation set goals and monitor progress toward achievina th^ •«,t.o 
Some data bases already exist that report on the sSad w^^^tJf 
umversities in emolling and graduaSj students m^tS^.^«^^^^ 



er!c 



AscipL ded among the existing data bases are the Integrated Postsecondary 

Educaoc ystem (IPEDS). the Recent CoUege Graduates Survey (RCGS), the Nationa 

Postsecc oident Financial Aid Survey (NPSFAS), and longitudinal data from the 
National udinal Study (NLS '72) and High School and Beyond (HSB '80 and '82) Ii 
four yean MELS 88 cohort of eighth graders wiU reach the age of eUgibiHty for atteiidini 
coUege. V, the exaction of IPEDS, each of these data bases is composed of a national 
representative random sample of students and is useful for estimating the demand by 
American citizens for mathematics and science curricula. Hiese data bases include race sex 
and atizenship distributions that also permit ethnic and gender comparisons within the U S ' 
Much work, however, is needed in order to make important international comparisons of the 
supply of mathematicians and scientists with the supply of teachers of mathematics and 
saencc. Parallel efforts are needed in developing ways to measure and compare the quaUty 
of coUege mathematics and science curricula and student achievement in American colleges 
and umvcrsiues compared with those abroad. Establishing cooperative data exchange 
agreements with foreign nations, comparable procedures and methods for coUecting data, and 
common language and definitions for describing degree levels and other outcomes are 
among the tasks that need to be accomplished. * 

The and information for higher education report cards should conespond to the 
objecnves of Goal #4 and to higher education's contributions toward that national goal 
Accordmg to the President and governors, higher education's most important contributions 
toward makmg America first in tiie worid in nuuhematics and science are as foUow: 

1. to increase by at least 50 percent the number of students who are trained to 
teach mathematics and science in tiie nation's elementary and secondarv 
schools; ^ 

2. to increase the ovenJJ number of American students receiving 
baccalaureaii i doctoral degrees in die fields of matiiematics 
science, ana en^.. , ui U.S. coUeges and universities; 

3. to increase tiie number of women. Blacks. Native Americans, and ffispanics 
who major in science, matiiematics. and engineering in coUege and increase 
tiie proportion tiiat tiiey rqwesent of tiie total harf^i^^irntr, master's, and 
doctoral degree recipients in tiiose fields. * 

In addition, zppropriatt assessment instruments (examinations) need to be developed 
for mtemational comparisons of America's coUege and univenity matiiematics. science md 
genaal education curricula and for measuring tiie matiiematics and science achievements and 
abilities of Amenca's coUege graduates-tiiose who major in matiiematics and science as weU 
as for coUege sttidents --neral. 

This wiU require intt , inal cooperation. coUaboration. and coordination. The U S 
Dqwrtment of Education shou ake tius effort a priority, and use tiie resources of its 
Office of Educational Research^ Improvement (OERI). e^iedaUy tfie Office of Research 
(OR) and tiie National Center for Education Statistics (NCES). toward tfiat end. ITie 
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o> using tne international compansons data and information coUected bv the it 
DeDartment of PAi/w»».„„ vuiiw.«w oy me u.5. 



Department of Education. 

Assessmftnt-. The Rpp^r^ r^ ^ 



September 100] 



conducS';?*!'??^ Posteecondary ^ucation Data System (DPEDS) is an amiual survey 
conducted by the U.S. Depaiiment of Education that can be used to provide baseline lOQl 

raraaiy, and sex in coBeges and universities. Stale and institutional conroarisons vrithin 
states a« also possible using IPEDS. This data base can also be to Zte tte 

U.eS^o'S; '^y^P°«> ^ ^ ^S, however^ nTSdfdata on 



The IPEDS does not provide data on the number of Americans or foieien nationals 
who enroU or receive theii degrees abroad, but it does include the^b^ TftLS? 

^J^i. govaiiors' report card to be complete, degrees awarded by coUeges ^d 
un^yeraties m Canada. Europe. Japan. Korea, and China, etc.. in mathenLics and ^wc 

mose onered m the U.S.. should receive some monitoring. This wiU leauire a cooottative 
arrangement between the U.S. and each of these other nations. cooperative 

The Division of Science Resources Studies of the Naitional Science Foundation 

SsS "-^ZT^. °i ""^T "^"^ Studen^aL PolS^rates 

(GSESP). TTie survey mdudes aU dqnrtments in science disciplines at aU U S coUeees 

physical, mathematcal. computer, agricultural, biological, social. engineeSr^rfidds 
^Ett^* lesults from this survey can be used to modtoS ii^f''''' 
emoUmwit m U.S. master's and doctoral degree piogiams by the sex. act/eOiJ^Ll 
SXt^hl'Sr?"^ TTiis survey also itports the'major sii^SdT 

assistance that students receive at these graduate sdiools. Like ffEDS howeva 
mtemational comparisons axe not currently available. 

fid^'tmrT. «™ <^ « mathematics, sdence. and engin«Sig 

fields fiiom U.S. umversities. These data are gathered via tiie Survey of Earned Doctorates 
(a jomt effort sponsored by the NSF and four other fedeial agendes); which «n^T 
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^uatestodrnt ing Ph.D.s in a given year. In addition, the Institute of Ihtemational 
Education (IE), /ate foundation dedicated to oiganizing and fadUtating international 
student exchanges nducts its Annual Census of Foreign Students (foreign citizens enroUed 
in U.S. institution . aU degree levels). The NSF and HE data bases constitute tiie most 
importtnt current lections of infonnation on foreign students enioUed in U.S. matiiematics 
and science prograrus, as weU as regulariy tiacldng tiic performance of U.S. students 
^^T^^i?'^* in these fields. Data are also coUected by die IPEDS surveys of 

thr U.S. Department of Education, based on institutionally reported information. 

Data concerning degree production and stiident achievement in matiiematics and 
scienw abroad are reported regularly by tiie national ministties of education of countries tiiat 
partiapate m eitiier tiie UNESCO or OECD data exchange programs. Hiese data are 
occasionally compared witfi U.S. data in NSF and otiier data bases, but tiie lack of refined 
OTtena and s^d^ds for direct comparison makes such analyses difficult and problematic. 
This wiU need to be corrected in order to produce r^rt cards on Goal #4. 

y^^^^ Education and Adult Learning Division of tiie U.S. Department of 
Education (OR/OERI) is currentfy engaged in a joint project wifli tiie NSF to develop a 
wding system for foreign institiites for tiie Survey of Earned Doctorates On which the 
Education Department participates). TTus project, led by E. Stqrfien Hunt, wiU result in a 
oodmg system tiiat aUows comparability between U.S. and foreign institiitions by level and 
types of degrees. While it is a start in die right direction, more such work is needed. 

Adequate measures forjudging tiie strengtii and quaUty of college and university 
matiiemadcs and science curricula or for international comparisons do not exist for tite 
^^^;i^°^^99l^eponcaid. Such measures need to be developed for titerqwrt cards in tite 
year 2000 and beyond. Hw science and matiiematics parts of tiie GRE are inamropriate for 
Has purpose because tiiey were designed for aiid are taken only by tiie sdect sub-pmulation 
of baccalaureate degree recipients who plan to attend graduate school; titerefore tiu content 
Of these tests do not reflect tite achievement outcomes expected of tiie general population of 
coUege graduates or tiie general population of science and matiiematics majors in 
undergraduate school. In otiier words, die results of tiiese tests do not rqaesent tiie overaU 
stasngtii of tiie undergraduate curricula or tiie science and matiiematics major field curricula 
UnUce otiier nations tiiat comprise tite comparison group for most international comparative 
sttidies m lEA and lEAP, examinations are not noraiaUy (currentiy) required of graduatine 
college semors from U.S. colleges and universities. 

The Major Field Achievement Tests, developed jointiy by tite Educational Testing 

Service and Graduate Record Examination Board in 1989, are multipleK^.sic- tests used^ 
sniaU number of coUeges and univenities for assessing tite achievement of coUege seniois in 
mathematics and science (biology, chemistry, and physics) curricula. Some of tiw science 
professional societies have also developed standardized multiple-choice tests for college 
seniors mooring in science disciplines, e.g., chemistry. While tiiese may be usefiil 
mstniments to build upon, tiiey may not sufficient for assessing progress toward achieving 
national goals or for international comparisons for tite following reasons: 
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1. They do not reflect an agreed-upon national consensus of the skills and 
knowledge that students graduating from U.S. colleges and universities with 
mathematics and science majors nor those who are not mathematics and 
science majors should acquire. 

2. They do not reflect a national consensus on the methods of assessment that 
college and university faculty believe should be used to demonstrate progress 
toward national goals. 

3. In their present form they do not lend themselves to international 
comparisons nor do they reflect an international view or agreement about the 
science and mathematics skills and knowledge that college graduates in 
mathematics and sciences, teacher education, or other disciplines should 
acquire by die time they graduate ftom college. 

Recommendarinn^ 

1. The IPEDS should be used to report annually the trends on degrees 
conferred by U.S. colleges and universities and that data orj U.S. citizens and 
foreign citizens should be better edited for accuiacy. 

2. The survey of Graduate Science and Engineering Students and 
Postdoctorates (GSESP) should be used to repon and monitor trends in 
graduate student enrollment in tiie nuuhematics and science disciplines by sex, 
race/ethnicity, and citizenship in U.S. colleges and univecsities. 

3. The president and the governors should establish an international working 
group tiiat will establish a working relationship with other nations to develop 
comparable data bases to the IPEDS and the GSESP so tiiat U.S. science and 
mathematics enrollments and d(^rees awarded can be compared with those of 
other nations. 

4. As part of the cooperative arrangement developed in recommendation 3, the 
President and tiie governors should create a higher education version of the 
national examination system tiiat permits collies and universities to measure 
botii tile strength of undergraduate science and mathematics curricula, and tiie 
attitudes, skills, and achievement levels of students giaduating fiom science 
and matiien*atics curricula in U.S. colleges and universities compared witii 
their counterpart science, mathematics, and teacher prqxuation programs and 
students in otiier developed nations, including assessments tiiat compare 
general knowledge and skills tiiat reflect college graduates (non-sdence and 
mathematics majors) from U.S. collies and universities conqnre with 
counteiparts in otiier nations. This will require intetnaticmal committees witii 
rq>resentatives f om participating nations to plan and develop these 
examinations. 
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5. A for evaluating the outcomes of intervention plans and strategies 
of U.S coUeges and universities should be developed to measure their 

effec Particular attention should be given to evaluating: 

A Retraining and providing continuing professional 
development to existing mathematics and science teachers 
through sharing of facilities, feculties, technology, and summer 
institutes, etc.; 

B. Developing better ways of assessing teachers and students on 
their knowledge of mathematics and science so that teaching and 
learning is improved and to facilitate entiy of more and better 
mathematics and science professionals into elementary and 
secondary school teaching; 

C. Producing early intervention strategies to young people to 
inform them on mathematics and science options and to 
encourage them to achieve in those areas. 

6. The organization re^osibte for licensing and certifying teachers in each of 
the 50 states should be re<|uested to provide data for each tqport card on the 
number of current mathematics and science teachers by race and sex who are 
certified and licensed to teach in the courses of mathematics and science to 
which they are assigned. The Council of Chief State School Officers has a 
state-by-state data base that contains these data (reported for the first time in 
March 1991). Second, die state organizations should also provide data on die 
total number of certified and licenses mathematics and science teachers 
(currently teaching and not teaching) by race and sex. Hiinl, the state 
organizations should provide data on the number of newly hired teachers in 
mathematics and sdeace by race and sex who are certified and licensed. 
Fourth, die National Center for Education Statistics should provide data from 
die Schools and Staffing Survey on die number of current mathematics and 
science teachers who have a college degree major in die subject diey are 
teaching. 



Conclusion 

No one should expect die achievement of die steps ouUined above to be easy , but diey 
wiU never be accomplished until a coordinated effort is begun. In die course of setting up 
die comparative information system needed by U.S. educators and policy makers, die 
important differences diat distinguish American higher education (and secondary prq»ration) 
from foreign counterparts will have to be noted and accommodated. In some cases, what 
occurs in die U.S. at die postsecondary level may be secondary education dsewhere; odier 

examples of dus kind will challenge die process. Meaningful assessment data, however, 
must be developed, including mtemational comparisons. Hie bold but so far isolated efforts 
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of agencies like NSF and OERI need to be buiit upon and coordinated in order to enable the 
nation to achieve the vital objectives of National Education Goal #4. 
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Comparative Aspects of Quality Assurance 
in Higher Education 



Edgar Frackmann 
Hochschul-Informations-System 
Germany 



Introduction 

QualiQr and quallQr assurance in higher education have gained new attention in Europe 
and elsewhere. In order to better understand this specific concern about educational quality, 
the question might be posed: Why is this worry about quality not similariy directed toward 
the research function of universities? 

To tackle this question one has to investigate how and whether the research function 
of universities is being scrutinized. The traditional German uriversity is a research 
university; despite the Humboldtian principle of the "unity of research and teaching," 
IfiS^Kil is the primary and prevalent function of the university, whereas teaching apptan as 
a by-product. 

If one accepts research as a service function directed towards an "audience" (to avoid 
the term "customer"), one might ask whether this audience is able and has the means not 
only to judge the quality of these services but also to put sanctions and rewards to work. 
Research indeed seems to be a field of societal services in which judgment, lewaids and 
sanctions, and quality control seem to be quite well established, even if one contends that 
research is a field that can only be judged adequately by the researcher community itself. 
Three main mechanisms account for the existence of qioUty control in research: 

(1) Research is perhaps one of the most prominent areas in which professional 
(self) control is evident. Whatever researchers do, as soon as they expose 
their research results to an interested audience by publication or whatever 
means of communication (indispensable to their research activity) they 
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exp .^search to the judgment of their peers. This pennanent peer 

jud^ ;lies and impacts implicit self-control mechanisms on the 

proi reseaxchen choose research subjects, methods for research and 

style. jlication acceptable to their peers and base their research on the 

result leir peers' prior research (cf. Hardy et al., 1983). 



(2) Peer review and peer judgment is a widely accepted mec'^anism used by 
research funding agencies for the allocation of public and foundation money. 
This kind of explicit quality control seems to be accepted as a valuable and 
reliable mechanism by the researcher community as well as by the general 
pubUc (cf. Meier-Leibnitz. 1989). 

(3) Finally, one has to take into account that government and company funds 
devoted to more applied research exceed by £ar the public and foundation 
research funds mentioned so far, which are supposed to serve the basic 
research activities of higher education researchers. Only by exposing 
thismselves to ever new quality control will the higher education researchers 
will attract government and company research funds. 

Not considering the deviations from the ideal in detail will lead to the conclusion diat 
professional self-control as well as external control of research quality seem to be well 
established and hence that there is no reason why quality assurance of research fhould 
become an issue of public discussion. 

For the education function of higher education institutions, however, the situation 
appears to be somewhat different As long as education is a by-product of research, 
educational quality seems to be guaranteed by die quality of research. However, with higher 
education entering the - iducxdoa, with an increasing public es^ectation directed 

towards higher educaaor. ? ... ■ professional and vocational education and with tiie 
programs to be shaped equally by die development of disciplinary knowledge and by labor 
market requirements, the quality of higher education deserves sqnrate and special attention. 

One nught ask a question about the research function of higher education: is the 
audience interested and competent to judge upon die quality of die educational services of 
higher education institutions, and does this audience the sanctions and rewards needed to 
influence educational services? 

Witiiout any doubt, the triangle of clious O-c, students, their parents and employers), 
fiSfiZS of the discipline Q.e., colleagues, die profession), and the society rqnesented by the 
government is to be identified as die audience interested in higher educariim quality (cf. 
Frazer, 1991). Governments in most cases do not base their allocation decisions on the 
quality of education (usually as a result of missing quality criteria and quality evaluation 
competence). Employers, once tiiey need higher education graduates and cannot shift dieir 
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demand to other tertiary or vocational education sectors, are bound to higher ec . ^on 
whatever the educational outcomes of the institutions are. Students, if demand exceeds 
available student places, have no choice among institutions. The peers in the discipline have 
no reason to judge their colleagues' teaching, as long as their self-concq)tion implies their 
role as researcher more than their role as teacher. 

To conclude with regard to the educational services of higher education institutions, 
an audience does not exist that combines the power, competence, and interest to care about 
quality. No wonder there is some concern about how and whether quality in higher 
education may be ensured. 

What is Quality? What j<j Quality Assurance? 

Referring to the fact that higher education provides "products" or services for an 
audience, it seems to be self-evident that these services have to meet the values, expectations 
and goals of this audience as well- as the values and goals of the service providers. 

Quality is connectoi to the achievement of goals, values, and expectations, matching 
the goals of service providers with those of customers. As mentioned earlier the 
stakeholders in higher fidualiQii are represented by the government, while the students, their 
parents, and the employers are the primary clients and the peers; the latter arc "located" on 
both sides, as service providers and beneficiaries. 

Three aspects of quality can be distinguished: 

The characteristics and specifications of the product or service as such have 
to achieve the goals of the stakeholders. 

Each of the product or service units being delivered by the organization 
under review (i.e., the higher education institution) should meet the quality 
standards according to the general characteristics and specifications once 
defined. 

The quality of products and services has static and dynamic dimensions. 
The quality of products and services implies their continuous adjustment to 
changing goals, values, and expectations. 

What is true for quality appU^ to quality assurance as well. 

Quality assurance has three dimensions: 

(1) Quality assurance implies the identification of characteristics and specification of 
the product or service in such a way as to meet the values, goals, and 
expectations of the relevant clientele. 

29 



ERIC 



32 



(2) Qu irance also means monitoring (to control) or ensuring a standard. 
Quality ds, once identified for the product or service in general, are 
fulfilled uously by each of the individual products or service units 
deiiveret. 

(3) Qualit; assurance finally requires fi continuous adjustment of products and 
services to the changing values, goals, and expectations of the stakeholders. 



Alternative Qualitv Assurance Mechanisms 

Quality assurance in higher education seems to reflect only one facet of a general 
organizational problem, i.e., the "problem of achieving cooperation among individuals who 
hold partially divergent objectives," per Ouchi (Ouchi, 1979, p. 845, cf. for the following 
section Frackmann, 1991). It is useful to review Ouchi*s framework for "organization 
control mechanisms." 

Ouchi raises the question: "What are the mechanisms through which an organization 
can be managed so that it moves towards its objectives?" (ibid., p. 833), and he identifies 
two different mechanisms Obid., p. 845): 

the market mechanism, in which external forces in a competitive environment 0.e., 
the consumers) take over the function of evaluating the organization's goal 
achievement and product quality. 

the bureaucratic mechanism, in which rules axe expliddy set and monitored, either by 
a system of hierarchical superiors or by functionally separated "quality control units." 
Monitoring takes place an tlie basis of pre-defined "rules." "These may be rules 
concerning processes to be completed or rules which specify standards of 
ouq)ut or quality" (ibid., p. 835). 

With r^ard to quality assurance, one might easily identify two pain of alternative 
mechanisms which should assist in providing a broader perspective of what is dealt with in 
this context of quality assurance and of what Ouchi conceives as organizational coordination 
alternatives. It is suggested here that one refer to the distinction between internal and 
external mechanisms on the one hand and implicit and explicit mechanisms on the other. 
This distinction will guide the course of remainder of this paper. 

Implicit versus explicit control mechanisms. Goals and standards to be achieved or to 
be the basis for the monitoring of goal achievement might be defined explicitly. Goal 
a ch ievement and the assurance of quality standards might be monitored explicitly and a 
posteriori. Explicit quality control always implies setting rules a priori and a "bureaucracy" 
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that checks reality against the pre-defined standards a posteriori. Bureaucratic mechanisms 
separate additional steps within the "production process." As a result, bureaucracy interferes 
with the flexibility of adjustments to changing consumer requirements. Implicit control 
mechanisms are less visible than explicit mechanisms. They arc not characterized by 
separate organizational control units into the process of production or consumption. Ouchi's 
clan mechanism and market mechanism are identified as implicit organizational control 
mechanisms. 

lOtQUal versus external control mechanisms. An organization delivers its products or 
services to an external environment Whether an organization wishes to or not, the quality 
of its products and services is monitored externally, either by the consumer market or, in the 
absence of market mechanisms, by a bureaucracy having the task of ensuring minimal quality 
standards. An organization might not wait and see that external "sanctions" endanger the 
organization's survival, but it will instead try to identify the consumer's requirements in 
advance and make them guidelines for internal quality definition and quality assurance 
mechanisms. These internal mechanisms might either follow the explicit bureaucratic model 
or the more implicit one. The distinction between internal and external mechanisms, of 
course, depends on how the organization's or system's boundaries are defined or perceived. 

If we use the internal-external and the implicit-explicit distinction as headers for the 
columns and rows of a matrix, we can identify the following in regard to Ouchi: 

the clan/pmfession/culture mechanism as being positions within the internal/implicit 
cell. Quality assurance mechanisms identified as internal and implicit are not visible to an 
external audience. Quality and quality assurance in this sense are not only based on the 
commitment of the clan or profession members, but also seems to be built on the 
external audience's confidence in the^e internal invisible mechanisms. 

the market mechanism as being located within the external/implicit cdl. Quality 
assurance within the market mechanism still is not based on cleariy-defined and visible 
quality criteria but rather on the aggregate of the individual's judgment and choice. 

the bureaucracy mechanism as belonging to both the internal/explicit and the 
external/explicit cell. The internal explicit model still relies on the peen' judgment and 
criteria but implies less confidence in invisible mechanisms as it nudoes quality assurance 
more apparent for external stakeholders. The external bureaucracy is pohaps the most alien 
in (he research university tradition, which has been based on the belief that only peers 
(internal audience) are able to judge higher education and research quality. 
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not ALTERNATIVE QUALITY ASSURANCE MECHANISMS 





Implicit 


Explicit 


Internal 


Qan 

Profession 
Culture 


Internal 
Bureaucracy 


External 


Market | 


External 
Bureaucracy 



This ftamework of quality assurance mechanisms seems to be veiy useful as a 
foundation of further comparative research. The remainder of this i»per will be devoted to 
fiirther elaborations on this matrix. Some of the most common concepts to be found in the 
literature on quality assurance will be analyzed spedficaUy while the management and 
governance of higher education will be examined more generaUy. 

Six concq)ts or pairs of concepts will be reviewed based on the intetnal/extemal and 
implicit/explicit ftamework: 

self-regulation and external steering 

evaluation for improvement or for accountability 

the role of evaluation criteria 

peer review and performance indicaton 

the role of ui;. — « and output-criteria 

funding higher eduu;. quality assessment 
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Sglf-Renilation and PT| f niffl Stggring nf ff igher Educarinj, 

QuaUty assurance might br ^nsidered as only one, although verv imnnmnt 

cOTiext as a whole. TTie more general concqMnaluation of this higher education nolicv 
problem often appears in the form of ±e self-regulation/extenal sSfSS,^^ 

Tlie classical model of higher education poUcy is again the imoliHt/int.™! «tf 
«giilat.on model that rdies on the confidence in%itter the indiSSiSSrS^ta^^d. 

mgher edwaoon pohcy seems to be government planning as a form ^etim^^X^ 

on higher education. Govemmoits. howS^er, a^^^S^t^ 
Uraitations m judgmg research and education i-nd often rely either on^iariSXh«^.m. 
P«T judgment, •"■eroleof peers in the steering of the hiito^uSti^t^^^ 

^^ZTT "J" » P«ri»nS^^n^^S**' 

mgaiiJzational umt under review, and as criteria for discussions baod m W^™^ 

~d^S;SiroLrf ^-.-ya'^bereg^L'r^iS'iS^X^idering 
uici4iciinaiiney still belong to the respecttve academic communitv Th«rmviJ^n« 

Sl^aTi^lTh^ ^"""^"^ " « accTtT'fJttr^^el 
always appears to be a separate and visible "unit" of quaUty assessment 

steering'^Ss ;^gtS:^ 



HGURE 2: ALTERNATIVE STEERING MODELS IN HIGHER EDUCATION 
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Evaluatior 



juntabilitv or for Improvement 



In c express a moderate reluctance against evaluation in higher education, the 
higher educ: oommunity of^ stresses the claim that evaluation should primarily serve 

the purpose . xountability. Some doubts might arise about the adequacy of this 

concq)tualizan :i of alternative purposes. 

Evaluation might be conceived as a "collecting" of information on the strengths and 
weaknesses of a product or service, i.e., on its quality. The question arises s^ut the other 
purposes of this information; should it be collected for the: 

monitoring of whether the product or service meets the expectations, values, and 
goals of the relevant clientele, 

monitoring of whether the individual products and services continuously meet these 
quality standards, 

monitoring of whether imxlucts or services have to be adjusted to changing values, 
ejqpectations, and goals, 

are the major aims of evaluation. Thus the main reason for evaluation is maintaining and 
adjusting, i.e.. improving quality. 

Accountability might be conceived of as delivering the information of the evaluation 
process, mandatorily or as a result of a certain pressure, to external agencies. What else 
would external agencies such as the government or the customers of higher education use the 
evaluation results for than to relate their decisions and choices concerning higher education? 
The information, if j^jpropriaiely used within the decision-making processes, might induce 
adjustment or improvement of higher education quality on its turn. 

The underiying distinction of the widely discussed alternative between evaluation for 
accountability and evaluation for improvement thus has been identiiied as a distinction 
between evaluation being used to induce internal processes of adjustment and improvement or 
to induce external agencies to seek adjustment and improvement via sanctions or rewards. It 
is a question of ownership of the once-investigated information on the strengths and 
weaknesses of higher education performances. In our matiix a distinction is made along tiie 
internal/external dime 
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HGURE 3: EVALUATION FOR IMPROVEMENT OR FOR ACCOUNTABIL- . Y 
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Criteria in Explicit F-vfljn ation PTnces?»?8 

In many countries and their higher education systems, procedures have been 
established for explicit quality evaluation. Explicit quality assunuice was defined earlier in 
this p^)er a^ a separate step in the "production process," most often undertaken by separate 
organizational units. Explicit evaluations also imply certain oiteiia. The difTerent role and 
the criteria used within different ^roaches of explicit quality assurance processes are the 
fod of this paragrs^h. 

(1) The weakest form of criteria used in quality assurance processes is within- 
institution quality control and improvement efforts. These efforts are made 
visible and communicated to the public, but quality control criteria are not 
made explicit. Taking into account that higher education not only has to 
achieve internal but also external goals, this kind of quality assurance seems to 
be based on public confidence in the institution's quality to consider these 
external goals and criteria. This is the form cf quality assurance expodiod by 
German governments (e.g. the government of the Land Noidrhein-West&len) 
from their institutions. 

(2) To guarantee some "externality" within die quality assessment die process 
might be based not only on intemal peers but on peers from the academic 
community in general (external peers). Still, goals and criteria along which 
achievements are judged, are not visible and remain more or less internal to 
the academic community. The main emphasis of external scrutiny seems to be 
die approach of Uie Dutch VSNU program review efforts. 
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(3) -al agencies or stakeholders might have some doubts about whether the 

aiteria for quality are aiiplied and whether the process of evaluation 
jality assurance is undertaken appropriately and in a idliable manner, 
aing to a certain set of generally applicable standards. As a consequence 
-nese doubts, the approach of meta-cvaluation is being implemented, i.e. 
- valuation of the evaluation processes. This endeavor may be undertaken by 
the peers of the academic community (such as the Academic Audit Unit in the 
British university system) or even by a governmental agency (such as the 
Inspectorate in the Nethedands). 

(4) The next point on the "continuum of criteria use" in expUdt evaluation 
processes is marked by the explicit use of pre-defined criteria. Quality as 
nientioned earUer always has somediing to do with goals and with achievement 
of these goals. The evaluation we have in mind here focuses on the 
congruence of previously-stated purposes with educational outcomes. TTiis 
means that the goals and quaUty standards of the unit under review Onstitution 
department or program) are taken for granted and as a basis for evaluation * 
This kind of evalt: . on ^roach is being foUowed by the American 
accreditation t 

(5) n» final approach to be mentioned here is the one where predefined and 
geneiaUy appUcable criteria form the basis of cxpUdt quality control conducted 
by external agencies. This might be the underlying rationale for the 
"contractual financing" of higher education. 

Whai these alternatives to quaUty evaluation approaches distinguish criteria we Mowed a 
Ime in our matrix finom t -Tjal/impUdt cell to the external/explicit cell. 

nGURE4: ALTERNATIVE CiUTERIA IN EXPUOT QUAUTY 

ASSURANCE PROCESSES w^/vj-iii 




Peer Review and Performant^^ Tnfli^^"^ 



Perfonnance indicators and peer reviews are regarded in the literature as very crucial 
opponents of quality assurance or quality assessment higher education. We will try to 
determine which of the dimensions of our matrix we might use to explain the difference 
between these two ^proaches for higher education. 

With peer review it is always difficult to grasp the underlying criteria of evaluation. 
The basis of peer judgment always remains somewhat invisible. Even external peers (i.e., 
peer groups that do not belong to the unit under review) apply the means of peer review in 
which criteria remain implicit. Performance indicators do not necessarily follow a totally 
different sqpproach from peer reviews. There are performance indicators representing nothing 
but quantified peer review results. The use of research unit rating in UFC funding 
mechanisms in the U.K. and bibliometric methods based on citation or publication counts in 
referecd journals bajdcally are peer review results. However, perfonnance indicators that 
reflect the quality of educational outcomes are quite different firom the ones mentioned so far, 
in that they take into account external judgment on the institutions* quality of the educational 
functions. Performance indicators showing the employability of graduates are based on 
external criteria and external judgment. Thus the more important distinc^on that goes 
beyond the distinction between peer review and performance indicators is the one between 
internal and external criteria used to assess die quality of higher education. 

In our matrix, while describmg the different species of peer review and performance 
indicators, we followed a line from the internal/implicit cell to the external/explicit cell. 



HGURE 5: PEER REVIEW AND PERFORMANCE INDICATORS 
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Input-, Process-, Jutput-onented Quality Assuiance Meehamsms 



Let us assL. i once more the case of explicit quality assurance mechanisms in which 
criteria are used. Certainly higher education has to deliver services to an external clientele. 
The evaluation of higher education quality might be restricted to the evaluation of the input 
and the process of higher education performances. This might be based on a confidence in 
higher education, believing that the "adequate" output will be produced if only the 
prerequisites for this ou^ut are ensured. OuQ)ut criteria in this community require explicit 
ouQ>ut criteria, whereas the alternative mode of evaluation focuses on educational outcomes. 

This distinction between input> and process-oriented quality assurance on the one hand 
and ouQ)ut-oriented quality assurance on the other is located in our matrix along the 
implicit/explicit dimension. 



HGURE 6: INPUT-, PROCESS- AND OUTPUT-ORIENTED QUALITY ASSURANCE 
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Funding Higher Educati on and Oualitv Assessment 

With regard to funding, higher education is totally dependent on external stakeholders 
or agencies. Whether members in the higher education community like it or not, quality 
always plays a certain role in funding. Funding relies heavily on higher education's meeting 
the goals, values, and expectations of the fimders or those represented by die funders, or in 
modifying external expectations in fiivor of its own goals and values. 

Two manifestations are imaginable in which quality oiteiia play a role in funding higher 
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education. The more implicit one is either the matkst mechanism or funding based on 
negotiations. With the market mechanism, quaUty criteria remain implicit Funding is the 
aggregate result of many quality-based individual decisions. Negotiations between higher 
education institutions and government often do not reveal quaHty criteria veiy clearly either. 
The more explicit funding mechanism is the one based on quality-oriented performance 
indicators. Funding criteria in this case must be well understood by both the funding agency 
and the fund recipients. In between these two alternatives one might locate the funding based 
on peer review, and the results of peer review being transformed into performance indicators 
(cf. the UFC ratings). 

In our matrix the funding alternatives based on quality assessment are to be 
differentiated along the implicit/explicit dimension. 

HGURE 7: ALTERNAITVE FUNDING MECHANISMS BASED ON QUALITY 
ASSESSMENT 
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rnnelusions 

Quality seems to be a broad concept related not only to the achievement of the higher 
education community but also to the goals, values, and expectations of external stakeholders. 
The quality of higher education is dealt with not only in quality assurance prxesses but also 
in more general issues like steering and funding of higher education. 
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r«nr*«J^f ^ * ftamework for clarifying alternative 

f„ ^"^"^ ""^ mechanisms that are cunently bckg discussal, 

analyzed in ature, or occurring in reality. » 

^rniJHt^'n f""^?" mechanisms on the one hand and 

cxphci ana . mechamsms on the other, it seems indeed to be possible to eSi Ae 

accountability or improvement, peer review and pe^ifbrmancc indicators, and the u« of 
input-, process- and output criteria. A comparative analysis is stiU needcxi of fte 
implementation of these alternative approaches in the higher education context 
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Questions of Quality in UK Hiyher Education 



John Brennan 
Council for National Academic Awards 
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The Quality Agenda: Higher Education in Britain 

Britain has recently and aptly, been described as "slouching towards a mass system" of 
higher education. The traditional dominance of the universities has been broken, at least in 
terms of student numbers, by the n^id expansion during the 1970s and 1980s of the 
polytechnics and other colleges. These two kinds of institutions now account for over half of 
all students. 

The participation rate in the UK is around 15 percent. Despite the variety of 
institutional types there is, by international standards, remarkable homogeneity in tiie 
educational product. The three- or four-year bachelor's degree prcdominaces in virtually all 
institutions. Certainly tiiere has been considerable curricular innovation in the polytechnics 
but it has been within the largely consensual framework provided by the acauwmic puiposes . 
and standards of the bachelor's degree. 

The maintenance of broadly comparable academic standards across the entire system 
may be one reason why it is only 'slouching' towards being a mass system. Many 
commentators have noted Britain's slowness in moving from elite forms, not just in higher 
education. In higher education, as Martin Trow and others have convincingly argued, tht 
setting up of the council for National Acadenuc Awards in the mid-1960s to oversee 
academic standards in the new polytechnics with explicit reference to standards in the 
universities, was primarily responsible for the preservation of the 'gold standard' of the 
bachelor's degree across the entire system. 

More recentiy tiie slogan, "fitness for purpose" and the growing importance attached 
to institutional mission statements have introduced a greater differentiation into the system, at 
least at the level of rhetoric and marketing. However, the assumption of broad comparability 
of standards is maintained. 

]^easuring and Assuring Quality 

Is comparability of standards actually achieved? Higher education, like most public 
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services, v m onto the defensive during the Thatcher administration. It was 

bombardec ^rformance indicators and other efficiency measures and the autonomy and 
self-T^ula nigher education professionals were increasingly challenged. 

The deal imperative towards qualify assessment has taken two main forms. First, 
repeated fiin . ag reductions posed the questioi<i, "what should be cut "? Part of the answer 
was the cutting of the low quality and the preservation of the high. The problem was how to 
distinguish between the two.l Attempts have been made to measure quality and to rank 
institutions in terms of a mixture of performance indicators (such as research productivity) 
and judgments of academic peers or government inspectors. 

The second form of imperative came firom an evident skepticism among members of 
the Thatcher government and its supporters about the standards current diroughout the entire 
public education system coupled with the conviction that "something must be done about 
them." The academic world was forced onto the defensive in relation to its central value, the 
maintenance and enhancement of standards. Questions of actually how institutions assured 
the quality of their work were voiced increasingly. At the end of 1990, the traditionally 
autonomous universities created under the umbrella of the Committee of Vice-Chancellors 
and Principals, a national A nic Audit Unit whose purpose was to ask this question of 
individual universities. 

The assessment of qualiQr in British higher education needs to be seen along a number 
of dimensions. Tliese axe summarized in the following diagram: 



Quality Aissessment in UK Highg r Education 

Quality Measurement Quality Assunmrft 

Performance indicators Peer judgments 

Institutional rankings Comparability of standards 

Differential funding Public accountability 

(External/governmental) (fotemal/institutional) 

Thus, qu ^"nrement involving the use of performance indicaton has been 
used primarily by gov , * agents to make funding allocations. Quality assurance, 

with a much greater empn.^ iudgments made within the academic community, has 

been used by higher education ins. jns to satisfy the growing demand for public 
accountability over quality and standards. National and institutional procedures and 
methodologies for assessing quality have been devised that focus variously <mi institutions, 
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academic programs, staff, and students. It seems that everything is being assessed by 
everyone. 

New Policies for the 1990s 

In May 1991, the British Government published a White Paper in which major 
changes were proposed to the higher education system, Uie most notable being the 
dismantling of the binary division between the universities and the polytechnics which has 
been a central figure of the system since the end of the 1960s. A single jfunding council, 
university titles, and full academic autonomy for the polytechnics are central features of the 
proposed new system. A higher education system will thus be created that is chanu:terized 
by expansion, diversity, innovation, and high quality. 

Considerable attention is devoted in the White Paper to the issue of quality. The 
assurance of quality is regarded as a matter for higher education institutions themselves with 
a Quality Audit Unit, owned collectively by the institutions, responsible for the audit of the 
quality assurance procedures of individual institutions. But in addition the funding coundls 
for England, Scotland and Wales will each establish Quality Assessment Units. These will 
assess quality for funding purposes and will inform the public, particularly students and 
employers, j^ut the relative quality of institutions and the ovoall quality of the system. 
Legislation is scheduled for the end of 2001. How the Audit and Assessments Units will set 
about their respective businesses is far from clear. 

Descriptions of two very different research projects into the assessment of quality 
comprise the remainder of this paper. The first project examines the development of a 
quantitative performance indicator of student 'value added'. The second is an attempt to 
compare quality across the higher education systems of different countries using primarily the 
qualitative measures of peer judgments. The first is an example of quality measurement, the 
second of quality assurance. The conclusion questions the sqparability of the two. 

Value Added 

Eariy in 1990, the CNAA was asked by the Polytechnics and Collies Funding 
Council (PCFC) 

to test different approaches to the measurement of value 
added, and . . . determine a methodology for the 
calculation of value added, based on a comparison of 
entry and exit qualifications relevant to the operational 
needs of the PCFC and higher education institutions. 

The work was commissioned to support the deliberations of a PCFC Committee on 
Performance Indicators. The briefing paper stemming from these deliberations was to 
provide advice on the use of performance indicators by the PCFC and institutions, 
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respectively, ii x>licy- making and planning processes and to suggest a range of 
perfonnance in -s that might be used to evaluate aspects of institutional performance. 



The xt brief implied a relatively narrow conception of value added: 
essentially a mea 3 of student achievement which took into account the effects at 
differential studer, .nputs (i.e., entry qualifications). The honors classification of student 
achievements on the bachelor's degree and the suppt sed comparability of standards across 
the system provided a largely credible ou^ut measure (one that was not wholly uncontested, 
however, along with its supporting assumptions). The generally lower entry quaUlications of 
polytechnic and college students, when compared with those of their university counterparts, 
needed to be taken into account in evaluating institutional performance in relation to student 
achievements. 

Several methr. - value added in terms of the relationship between 

entry and exit qualific . doped in the UK. Their problem was that they 

were based on arbitr^ : sumptions about this relationship. Indeed, as the project was to 
show, they generally gave too much weight to entry qualifications so that a high value added 
score was unduly determined by ]q:k entry qualifications. 

The comparative value added measure (CVA) developed by the project avoided 
this difSculty by using the actual relationship between entry and exit qualifications, as 
indicated by national data held by the CNAA, as the base against which an expected value 
added score could be calculated for an individual institution or any sub-group of students. 

The CVA compares the exit results expected for students with particular entry 
qualifications with their actual exit qualifications. It allows an assessment of a course's 
progress and success given entry profile. The method can also reveal whether 

iqjparent overall success coiicciu;* icuuve failure for students with certain entry 
qualifications. 

The report of the project (The Measurement of Value Added in ffighw 
Edlicalion, PCFC/CNNA, London, 1990) has aroused considerable interest. Tlie CVA 
appears to have attained a credibility-at least in the research community-which no other 
value added method has achieved. Politically and institutionally, however, the response has 
been more mixed. Precisely because it is based on non-arbitrary assumptions, it is less open 
to manipulation for particular purposes. 

Litemational Comparison nf Qualify 

The second project is currently being undertaken by the CNAA Quality Support 
Group in collaboration with the Center for Higher Education Policy Studies in the 
Netherlands and the Hochschul Information System in Germany. Its aim is to assess the 
possibility of making valid and reliable comparisons of the quality of higher education in 
several European countries. 
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The rationale for the project is found largdy in the imminent arrival of the single 
European market and the consequent increase in the mobility of skilled labor and of students. 
These pose issues of comparability among the courses, institutions, and graduates of the 
various national systems of higher education. 

The approach to the assessment of quality being adopted by this study is one of 
peer group review. The aim is not to rank either institutions or systems. Because quality 
must be related to system or institutional goals, international comparisons of quality cannot 
be hierarchical but should be descriptive of the different facets of qualiQr and standards to be 
found in different countries. 

The study is currently in its pilot phase. This has two stages: (i) the collection 
and analysis of descriptive information about systems and courses, and (ii) intemadonal peer 
review of selected courses. 

Ten economics programs drawn from the three participating countries were 
selected for the pilot phase. They provided details of their curriculum, teaching and 
assessment methods, students and staff. This information was considered by a peer group of 
nine economists, also drawn from the three participating countries. Discussions also took 
place with faculty representatives of the ten programs. 

The results of the pilot study will be reported in due course. Strengths and 
weaknesses of the programs from the different countries were identified by the peer group. 
However, the peer group was more confident in its judgments about the content of programs 
than about the learning outcomes arising from them. If the aim is to compare the qualities of 
graduates rather than the qualities of study programs, then a complex range of input and 
environmental factors needs to be given greater consideration. 

Extension of the work to cover other study programs and to induce other 
countries is being considered. 

Some Conclusions 

These contrasting examples of research projects on quality assessment being 
undertaken by the CNAA Quality Support Group indicate the range of questions being posed 
under the quality agenda. Although they support the distinction made earlier between quality 
measurement and quality assurance (the former using performance indicators to make finding 
allocations, the latter using peer judgments to meet public accountability for standards), such 
a distinction is in practice very blurred. 

The conclusion of the value added report~and other reports (m the use of 
performance indicators-was Uiat such indicators could aid but should not r^lace peer 
judgments. The international peer review of economics will make use of considerable 
amounts of quantitative and other performance data as a basis for judgment and evaluation. 
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The point can further be illustrated by the tendency~in the UK at least~to convert 
qualitative judgments about teaching or research into quantitative indicators or rankings to 
support funding allocations. 
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rnnriT^tualizatmn nf College Quality 



Jeffery Gilmore' 
U.S. Department of Education 



Introduction 

The purpose of this conference is to develop an agenda for international 
comparative research in the areas of accreditation, self-study and assessment, 
programmatic and curricular review, assessment of student learning, and math and 
science teaching and learning. The ultimate goal of these endeavors is the development 
of comparative measurements of quality in postsecondary education that will endure 
under the light of international examination. While this lofty goal is one that I 
wholeheartedly support, I am somewhat concerned that we may be starting from 
several different conceptions of what is meant by quality. In £act, I am aware that 
addressing quality issues from a variety of perspectives may be exactly what the 
conference organizers had in mind and that a "Tower of Babd" appnnch may be 
precisely what is needed to cut through our paradigmatic and cultural blinders and gain 
enlightenment. However, lest we wish to M victim to the same fiite as those in^ous 
citizens of Shinar so long ago, I offer up these humble thoughts on conceptualizations 
of college quality. 

Measuring Educational Quality 

One of the most persistent problems facing colleges and universities has been 
the difficulty of measuring and explaining educational quality. There are at least four 
general approaches to the definition of institutional quality: by outcomes, value-added, 
resources, reputation and selectivity, or by some combination of these^ Even where 
there is agreement on defining quality, great obstacles remain to measuring it, in part 
because many attributes are not easily quantifiable, and in part because quality, like 
value and worth, are largely "in the eyes of the beholder." 



'This paper was prepared by Jeffery Gilmorc in his private csqadty. The views are those 
of the author and no official support by the U.S. Department of Education is intended or should 
be inferred. 

^Alexander W. Astin, "Why Not Try Some New Ways of Measuring Quality?" EdttCation 
Record 63(2): 10-15 (1982). 
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- Quality. The most common and time-honoied approach is to ass es s 
educati( iity on the basis of institutional resources. In economic terms, the 
"inputs' le system have defined quality. The inputs into an educational 
institutic lough complex, are the easiest to measure as they arc most often 
expiessec quantitative terms. Inputs include such resources as capital, labor, and 
equipment. Using this approach, educational quality is defined in terms of faculty 
salaries, the number of faculty with Ph.D.s, faculty-student ratio, library volumes, the 
size of institutional endowments, the square footage or value of the physical plant, and 
the like. A complicating consideration in the measurement of inputs are inputs other 
than those measured by costs, including the personal attributes of students, fiiculty, and 
staff. These include student ability and effort, faculty energy and expertise, and staff 
commitment and skill, to name but a few. Intangible institutional inputs include 
environmental synergies and cultures that motivate and facilitate educational outcomes. 
Also not to be discounted are family and community support, encouragement, and 
expectations of success. These inputs contribute to the educational enterprise. They 
also make the measurement of quality a very complex undertaking. 

Reputation and Selectivity Another s^roach has been to measure educational 
quality on the basis of institutional reputation and selectivity. Together with the 
documentation of resources and the prices charged for tuition, institutional r^utalion 
and selectivity form the grist of the many rankings of quality found in popular 
magazines and college guide books. Parents and prospective students easily grab onto 
this information and note the wide ranges in average freshman Scholastic Aptitude Test 
scores On 1987 they ranged from 800 to 1,440), tuition charges (ftom $3,991 to 
$11,880 for private colleges), and acceptance rates (ftom 15% to 98%). Many 
colleges stake their future on these scores and rankings, and fiunilies often maif*^ 
critical educational ; :: r -n5 on them as well. However, measuring educational quality 
and value is much ' plex than those comparisons of tuition charges, average 

SAT scores, acceptance rates, and otiier components of an institution's reputation. 

Qualitv of Outcomes and Pmdnrts More recentiy, the measurement of 
institutional quality has focused on institutional ou^uts or educational "products." 
Measuring outputs is problematical. First, there are tiiree major ftmctions of higher 
education-teaching, research, and public service-and each has a different set of 
products. Moreover, tiie matter of what constitutes a "product" of higher education is 
in some disagreement. Some view it as learning in aU its manifestations, including 
knowledge creation, transmittal, and preservation, and changes in peq>le's knowledge, 
characteristics, and behavior. Otiiers focus on such tangible "goods and services" as 
credit houn, degrees, publications, discoveries, and public contact hours. Still otiiers 
consider outputs to be synonymous with such short- and long-term outcomes of 
education as increase: ^'me earnings and job status. An additional difficulty in 
measuring tiie quality Oi ^>nal products in higher education is tiiat different types 
of colleges and universities. ssions Uiat emphasize tiie teaching, research, and 

public sendee functions to varying degrees. Diverse missions make comparisons of 
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outputs among institutions problematical, if not insupportable. 



A promising outgrowth of this new tradition has been the recent explosion of 
interest in the assessment of student outcomes resulting from the educational 
experience. Using this approach, thr, measurement of quality might include an 
assessment of content learning in the disciplines (theories, methods, and knowledge) or 
the development of specified cognitive skills (such as critical and analytic thinking, the 
ability to synthesize material and ideas, and the ability to draw and defend 
conclusions). Quality measurement might also include an evaluation of student growth 
resulting from the educational experience (including moral, physical, social, 
psychological, aesthetic, and emotional development). Or, quality might be revealed 
through an assessment of student outcomes, both diiect and indirect, resulting from the 
collegiate experience (including degree attainment, further education, employment, 
lifetime earnings, socioeconomic status, civic involvement, personal lifestyles, and 
satisfaction with life). 

Retum-on-Investment an d a Value-Ad ded Approach. Enthusiasm for the 
outcomes approach outlined above and the large number of studies that have attempted 
to evaluate the consequences and performance of education must be tempered by two 
important considerations: the investments made in education (including money, time, 
and effort), and the differential abilities and bacl^unds of students entering college. 
The investment-in-education consideration comes from human cs^ital theory-the idea 
that investing in people's personal development will reap social as well as individual 
rewards such as better citizenship and higher levels of production in the w<nrkplace. 
Studies have measured the effects of education on such things as job satis£[iction, 
personal health, political socialization, criminal behavior, marriage and divorce, the 
ability to perform complicated tasks and adopt to changing conditions, economic 
growth, lifetime earnings, and occupational distributions. However, past studies that 
have focused solely on outcomes have several shortcomings in the assessment of 
educational quality. First, researchers must be sure not to limit analyses to quantitative 
outcomes that give no information about qualitative effects. Second, studies must 
disaggregate data by institution attended since it is reasonable to expect that the 
economic, social, and political impacts of the educational experience vary considerably 
depending on the particular institutional setting'. Third, outcome studies must control 
for students* native ability and other personal background factors. Education alone 
may or may not make people more productive workers or better citizens. Fourth, 
labor market conditions can also affect graduates' earnings. 



'Morton Owen Schapiro, "The Concept of Productivity as Applied to U.S. Higher 
Education," paper prepared under contract to the U.S. Department of Education for presentation 
at an Organization for Economic Cooperation and Development (OECD) conference on "The 
Concept of Productivity in Institutions of Higher Education," May 25-27, 1987. 
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An :>T7iative hypothesis for why higher wages are often associated with iiigher 
levels of ed uonal attainment, or why one institution's gnduates do better than 
another's, is it education serves as a screening system for society. That is, one 
principal fimcuon of education is to screen for individuals with the greatest motivation 
and innate ability. Students not only vary considerably in their academic ability and 
potential at the time of matriculation, but, and more to tiie point, different colleges 
attract and recruit different types of students with respect to student intellectual 
disposition, emotional temperament, motivations, attitudes, values, goals, educability, 
openness to change, and potentialities for development Various studies over die yean 
have well documented the differences in student characteristics and the fact that these 
differences vaiy considerably, not only among students within an institution, but, more 
importantiy, between institutions to a significant degree. Collies, tiuough the 
admissions process, identify only the best academically prepared students, and 
institutions witii die highest selectivity requirements have the best students. Differences 
in lifetime earnings and o'J - - romes may, therefore, reflect the fact that colleges 
merely identify die mor- nis ratiier than increase tiieir skills. The "better" 

colleges may not provide a. oetter eaucation but rather, may serve to identify and dien 
"certify" the better students. Thus, a refinement of the outcomes s^jproach to the 
assessment of quality incorporates a consideration of the "value-added" by the 
educational experience. 

Conclusion 

The quality of undergraduate education is an issue diat promises to be highly 
visible for some years to come. As we move aggressively to assess student educational 
outcomes, we need to work from a research-based concq)tual framework that helps us 
to fame die right questions and interpret the results of our efforts. I hope tiiat 
consideration of the conceptual sqn>n)aches to measuring institutional quality presented 
above will be helpful. I also hope tiiat, in our rush toward universal assessments and 
institutional accountability standards, we don't extinguish die distinctiveness and variety 
of educational missions pursued by our nation's collies and universities. That would 
be a tragedy indeed. 
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Taxonomy an d romparison of Approaches 
tft 0^p»^y Assurance jp Higher Education 

David Woodhouse 
Hong Kong Council for Acadeniic Accreditation 



Let us define quality assurance (QA) in higher education (HE) as mechanisms and/or 
processes designed to ensure that degree courses are of an appropriate qualiQr, that is, that 
the programs, admission levels, context, and resources (human and material) of the courses 
are such that graduates are likely to achieve certain standards in terms of knowledge, skills, 
understanding, and abilities. 

It is assumed that the QA mechanism also will permit some sort of objective statement 
about this quality in relation to other institutions. 

In some countries (such as the UK, Australia), some institutions (universities) have 
been totally responsible for their own quality assurance (internal assurance) while others 
(polytechnics, institutes, etc.) have been subject to external quality assurance pros^s^. This 
ample (if unequal) situation is now changing rapidly, and not in one consistent direction. 
Some institutes are being given more independence, for example, whereas some universities 
are being subject lo more external control. 

One reads monthly about a country's establishment of a body charged with ensuring 
the quality of its higher education. Naturally, there are nany different approaches to this 
task, and a question that typically arises relates to the efficacy of the various {^roaches. 
Presumably not all methods are equally good, and it would be useful to be able to determine 
which aq)proaches to QA are best, although this immediately requires a definition of "best" 
We are speaking, of course, of the quality of quality assurance bodies. 



Stage 1: Taxonomv 

To focus our thinking, we may attempt to identify and categorize the various types of 
approach. The following is an initial, three-dimensional, proposal for such a categorization. 
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1. nent department 

2. Su ry body 

3. Voluntary association 

4. Tertiary institution (whether responsible for its own quality only, or for that of 

some other institutions also) 

It may be that professional societies should be included here. 
D2. Scope 

1. A single course/degree'- m 

2. A whole discir .a or across institutions) 

3. Faculty/school 

4. Whole institution 
D3. Authority 

1. External, with 'teeth' 

2. External, advisoiy 

3. Internal 
Stage 2: aassificatjffn 

There are certainly examples of all of these (1 1 or 12) divisions. Further 
investigation would reveal whether there are instances of all of the 4 (or 5) x 4 x 3 « 48 (or 
60) cells of the three-dimensional array, whether categories have been omitted, and whether 
categories can be combined (e.g., on the grounds that thdr effects are the same). 

The task at this stage would be to identify as many QA bodies as possible, slot them 
into the cells, and revise the taxonomy as proves to be necessary. Prima fecie, the 
distinctions being made in this taxonomy appear to be real ones, and it may be that the QA 
bodies in any one cell perform similarly, with similar effect. 



} 
} 

} To the institution 

} 

} 
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Stages: Definition pf Effectiveness 

At this stage, choose a sample of the QA bodies identified, selecting no more than 
one from each cell. Its method of operation should be investigated through reference to its 
stated procedures and objectives and at least one entity (from dimension D2) for which it is 
responsible. It is clear that the effectiveness of a QA body must relate to the quality of the 
entities in its ambit. However, using the results of assessing an institution to draw 
conclusions about the QA body would be quite a difficult process. Furthermore, the 
activities of a QA body are only one factor in the quality of an entity for which it is 
responsible. It is necessary at this stage to gain some appreciation of the other factors 
involved, and their relationship to each other. Conversely the quality of its institutions is 
only one aspect of the effectiveness of a QA body. 

Effectiveness could be related to questions such as 'what does the body do to improve 
quality for students and staff, and .what evidence is there of your success?*; or to questions 
such as 'when did the body last close down an institution (or faculty or degree course)?' 
Such contrasts indicate that effectiveness dq)ends on the purpose of the QA activity and who 
has the power to set the agenda for it. Quality is largely a social construct, which suggests 
'iiat it is more j^propriate to identify the best match between the environment and quality 
measures than to try to define what is best in an absolute sense. 

With this caveat, we may need to address what is almost a second-level taxonomic 
problem. Having identified what is meant by 'quality' of an entity, can we 

i) discard those quality factors that do not derive from or dqpend on the QA body; and 

ii) identify any other desirable features of a QA body that should be incorporated in the 
definition of its effectiveness? 

Stage 4: £valualiQQ 

At this point, we investigate and compare as many QA bodies as possible, to see 
which are working effectively, in terms of the discussion developed at Stage 3 
(acknowledging that context of each QA body must be taken into account). It would be 
necessary to compare bodies finom different cells and also from the same cell. There may be 
more difference between bodies in one cell than across cells. This would lead one to look 
for other key differences between bodies. 

Conclusion 

Which QA bodies are most effective and in what senses is this the case? Which QA 
^roaches in the taxonomy give rise to these effective QA bodies? Gearly there is not 
global definition of "best." However, it may be possible to see what works where, and why 
it works, and whether it can be ^lied elsewhere. 
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Using Student Persistence Rftj«arch to Strenythen the 
Quality of Teaching and Institutional Processes 



Robert C. Fioh 
Syracuse University 



Introduction: Assessment for Quality 

In looking at the higher education scene both in the United States and internationally, 
many concerned with management and policy often focus more than they intend to on 
assessment for accountability and comparability. While these purposes in^ire competition 
for excellence, many impro vements in higher education are the result of ongoing 
collaboration among facui.y, administrators, and students on a r^ular basis to identify and 
respond to concerns r^arding quality. 

Themes: Points Toward Improvement 

Several points critical to encouraging continuous improvement of quality in teaching 
are addressed here. First, although assessment for improvement is not a new ooacepu some 
of the important dynamics utilized in assessment are Mdy new (such as a stronger emphasis 
on qualitative methods and procesnng arismg from Deming's total quality management 
[TQM]). Second, sufficient and clear assessment questions and outcomes emanate from a 
continuous concentration on incorporating multiple perq)ectives (such as those coming ftom 
different acadenuc units, institutions, and cultures). Third, the use of faculty, administrator, 
and student teams (botii separate and integrated) provides die engine upon which tiie TQM 
planning, doing, and evaluating and acting cycles related to assessment are accomplished to 
provide continuous improvement Fourtii, a continuous balance of methods incorporating 
both quantitative and qualitative data compensates for die inherent weakness of any one 
metiiod. Finally, if assessment efforts fail to have an impact on decision making, the 
assessment movement will end. 

Overview: Two Studies in Context 

In tius papa we nune Syracuse University's activities in die area of internal 

assessment priktiarily relau.. issions and retention analysis, and contrast this with work 
being dcme at die University oi ... .3urg in tiie Nedierlands as represented by tiie article, 
"Explaining Differences in Drop-outs Between Faculties in Dutch University Education,' 
autiiored by Riekde J. Bijleveld (1991) and distributed by the Center fDr Higher Education 
Policy Studies (CHEPS) at die University of Twente, die Nedierlands. These studies will be 
placed in the larger context of an increasing range of assessment efforts on the national and 
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international levels. 



Context! Syracuse University 

Syracuse University is a private research and teaching institution founded in 1870. 
Syracuse sits on the boundary between the east coast and the midwest, two regions with 
slightly different cultural contexts. Some of the traditions of both midwest land-grant 
il^tutions and east coast institutions have developed (see Boyer, 1990, for contrasting 
traditions in the development of higher education in die United States). The university was 
originally supported by the Metiiodist Giurch and local entrepreneurs of Syracuse who 
tiiought the thriving city needed a premier educational institution. It has never been a rich 
institution, but a hard-working one that is 55 percent dependent upon tuition dollars. 
Through intensive marketing efforts in the last five years, Syracuse has strengthened its 
endowment. However, it will never catch up to the fmuLraising machines of the most elite 
institutions in the United States. Given its high tuition, Syracuse does have a number of 
wealthy students (who are both visible and felt), altiiough 65 percent of the student 
population receives some financial aid. The institution has developed some very strong 
programs, particularly witii the S.I. Newhouse School of Public Communications (primarily 
at die undergraduate level), and die Maxwell School of Public Affairs and Administration 
(focused at the graduate level). Many other programs are of average to excellent quality. 

In his opening convocation to new students and dieir parents, the new Chancellor, 
Kennetii Shaw, described Syracuse as a private institution dedicated to public service. Tliese 
contrasts are very much a part of this mid-size (large for private universities) and diverse 
institution. 

As it pertains to this presentation, private research institutions in the United States 
have a unique advantage when building assessment processes for omtinuous improvement. 
""Private** means tiiat funding is somewhat independent of government, so Syracuse can forge 
its own path as it reads the environment and maintain its traditions. "Research** means tiiat 
Syracuse constantiy strives to develop faculty at the cutting edge of tiieir fields. In relation 
to assessment for continuous improvement of the management of higher education, there is 
considerable freedom of movement to forge an management research agenda. Faculty have 
the minds and power to focus on cutting edge issues. They can determine and define tiieir 
research priorities and foci. This assessment process at Syracuse is part of what makes the 
university very exciting. Faculty are not caught up in die typical drudgery of producing one 
report after anodier to serve various external constituencies, as are those at public institutions 
in die United States. 

The Office of Evaluation and Research at Syracuse University, which supports 
Campus Studies, is unique because of the nature of the institution. An official Office of 
Institutional Research (IR) was abolished IS years ago because the office produced reports 
deemed sufficientiy useful without sufficient consultation with faculty and administrative 
constituencies. The Office of Evaluation and Research has grown over time to bridge the 



55 



ERLC 



gap left jsence of an IR office. As OER has developed over time it has tried to 

respond iformation needs of decision makers. Its work is also closely tied to the 

academit. of the institution, as the Office is part of the Center for Instructional 

Developr ^ longstanding center dedicated to the improvement of undergraduate teaching. 

Retention: Tne Bread and Butter Issue 

Retention and the study of attendance patterns are the bread and butter issues; for 
higher education because they tie academic programs to resources for the teaching mission of 
higher education. For the teaching mission, the desired outcome is students graduating with 
academic and social achievements. The inputs are high quality students, and the resources 
(in money and faculty) to work with these students. 

Retention is an issue that cuts across institutional contexts veiy effectivdy-contexts 
like different countries, cultures, ^nd types of institutions (e.g., public versus private in the 
U.S.; public versus technical ^ ois in britain and other countries). It is the outcome that 
no one can argue with in tern . :£ its importance. If the purpose of assessment is to improve 
programs, it is the clearest indicator of what characteristics of a program work wdl and what 
needs to be improved. If the purpose of assessment is to account to external audiences or 
customers (such as governmental agencies), it is the clearest measure of program impact. 
All of higher education wants to deliver efficiently as many high quality students to society 
and to the work force as possible. Some public institutions have too many students and need 
to design programs that will work with these numbers. Most public instituti(ms do net have 
control of the resources for educating their students and need to show how quality decreases 
as student load increases, ^^cularly given dq>leting £vmlty and program resources (e.g., 
Austria has this probler as many public institutions in the United States, such as the 

State University of New : zt. Mm). In this era of greater accessibility to higher 
education, most institutions face the problem of producing high quality graduates despite 
increases in student diversity and skill levels. 

Retention is the ultimate performance indicator for assessment regardless of the 
purpose of assessment. Most other indicators are by nature less sufficient. They involve 
more value judgments (regarding the definition of high quality education) that dq)end on 
specific contexts. Retention as an outcome measure £uls somewhat whoi the goal of the 
process is to select small percentages of students who will complete their degrees (as is the 
case in France and some other countries). However, even in this context attendance and 
retention studies are still useful in determining the characteristics of those students who 
succeed and those who fau. 

Primarv Studies: Syracuse University a nd the Universitv of limburg 

Two studies will be contrasted to demonstrate the points mentioned at the beginning 
of this paper. 



gj^^ BEST COPY AVWUBLE 



56 



\ 



Primary Studies: Syracuse University and ttie University cf limburg 
Two studies will be contrasted to demonstrate the points mentioned at the beginning of this paper. 


Syracuse University Study 
(Tinto & Froh, 1991) 


University of Limbusx Study 
(Bijlr/eld, 1991) 


> internally onvcn insututionai asscssnieni 


^ cxicnuuiy unvcn wumpaiauvc asscMinciii 


► primary purpose is to improve program 
outcomes 


► primary purpose is to make institutions more 
accountable for their outcomes 


► assessment processes are as important as study 
conclusions 


► study conclusions hopefully motivate 
institudon-spedfic investigations 


► contrasts between subpopulations (one 
academic unit contrasted against others) 


► one institution contrasted against others, or 

contrasts between different types of institutions 



► data is a balance of qualitative and quantitative, 
explorative definition of performance indicators 


► data is primarily quantitative with precise 
definition of performance indicaton 



These two studies Fq)resent the best of what assessment has to offer in bringing 
accountability to, and fostering ongoing improvement in, higher education. Both avoid the common 
pitMs of assessment efforts. Internal assessment studies somedmes: become too narrow in focus; 
don't involve multiple perspectives; don't develop valid instrumentation (use nationwide or poorly 
conceptualized instruments); and depend too strongly on quantitative metiiodologies considered to be 
"state of die art," but which are limited in tiieir a^licability (such as some a^lications of repcaicd 
measures path suialysis studies). Unlike the Limburg study, many other externally driven 
comparative assessments lack sufficient focus; provide potential to motivate institutional 
investigation (parUy because every institution diinks th^ are unique); group institutions into 
categories diat are too general to be helpful and lose meaningful variations between institutions as a 
result [the Higher Education Data Sharing Consortium and die British study (Goedegd)uure et al., 
1990) combat this by identifying institutions]; lack sufficient focus on substantive differences 
between subpopulations wiUun an institution (such as contrasting different schools, different 
students, or different levels of student involvement in academic life). 
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The Edge: 



rsus Private 



Th. i study has one additional advantage in being the product of a private institution. 

As stated iiat this has meant for assessment is an increase in the drive to conduct studies 

for improvi. rograms rather than for accountability. However, in the next round of 
accredation r. vs (SU is reviewed by the Commission on Higher Education Middle States 
Association oi olleges and Schools), Syracuse must demonstrate that good assessment systems are 
in place. We \vill be able to show that our assessment systems for the purposes of improving 
p.x)grams are excellent; partly due to a focus <m maintaining an enrollment of high quality students 
(tuition dollars sccount for about 80% of the revenue stream). Faculty and administrators at the 
iiutitution have conducted studies to improve programs to recruit and retain high quality students. 
We will be able to demonstrate that our assessment systems address accountability from the 
perspective of looking at the interests of students, parents, alumni, and friends as customers who 
support the institution with tuition dollars and contributions. Accountability to government and 
other interest groups is minimal. 

The Harvard Studies initiated by the President, administered by Richard light (1990), are 
anotiier example of the type cf internal assessment that develops out of a private American 
institution context where i.e focu^ ^ends to be internally motivated. These studies are conducted by 
faculty, administiators, and students to understand the needs of students and faculty and to improve 
programs. In contrast to the Syracuse Campus Studies, which have been conducted by 
administrators witii funilty suj^rt, die Harvard Studies have been conducted primarily by faculty. 

Looking at the private institution context in the United States should prove useful to 
institutions that strive to make their assessment efforts more internally driven and focused on 
students and faculty. Li contrast, some of the ass^sment studies cunenUy being conducted in the 
Netherlanc*: and Great ~ : - die best examples of externally driven assessments focused 

upon accountability tc anding sources. This is not to suggest that the assessment 

studies in die Netiierlands ana ureat Britain do not respond to internal institutional audiences, but 
that their first priority is to external audiences. (Note: As we do get funding for minority students 
programs and for research, Syracuse and other private institutions are required to conduct some 
studies to respond to external audiences.) 



Conclusions: An Assessment Typology 

Higher education institutions and systems must attend to both purposes of assessment: 
continuous improvement for programs, and accountability to external customers. However, the 
influence and clarity of particular reference points, both internal and external to the institution, 
create die demand for certain types of information and die corresponding assessment studies to 
provide that information. 
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Purposes | 


Reference Group 


continuous imnrovement for 
internal customers (students and 
fiaculty 


accountability to external customers 
(funding sources) 


Internal 

► faculty 

► administrators 

► students 


Harvard Studies (Ught, 1990); 
Syracuse Campus Studies (Tinto 
& Froh, 1991) 


Limburg Study on retention 
(Bijleveld, 1991) 


External 

^ tvn^ c%f inctitiiHnn 

► private 

► public 

> cultural context 

► U«S. states/regions 

► European countries 


oiUQies cunQUCico uy nr«i/oy 
Carnegie Commission Reports 
(Boyer, 1990); Peer Review 
Systems, the Netherlands 


— T. ~: 

(Goedegebuuie, et al., 1990) 



Here we have attempted to define the contrasts and create a typology for the purpose of looldng to 
examples and particular contexts to expand the scope and quality of assessment studies. With this 
expanded scope and quality, institutions will improve more ra^dly, and internal and external 
customers will participate ever more so in this improvement process. In addition, the basic 
principles and processes of higher education will become clearer and more fully articulated as 
internal and external reference groups compare and contrast the issues they £ace within and across 
various cultural and historical contexts. 

Attendance and retention analysis cuts across cultural and historical contexts, providing one 
of the more promising focuses for assessment studies. The outcome is very concrete-retenticNi and 
graduation of high quality students. The connection of this outcome to funding is one of the most 
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stiaightfor 
of funding 
Information 
like the neec 
compl^ty 



. is extremely important as higher education is called upon to find new sources 
.aps increasing tuition, financial aid, and contributions from private business). 
Jiese studies will serve the changing external customer base of higher education, 
uild stronger relationships with business, and to respond to the increasing 
jucational and vocational needs. 
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An International Persnective on Assessing 
Baccalanreate Program Oiitcnmes' 

Trudy W. Banta 
Homer S. Fisher 
The University of Tennessee 

Structural differences among the systems of higher education in Western nations are 
sufficiently great that comparison of the systems sq>peais, at least initially, to be a futile 
exercise. Further examination, however, reveals some fundamental amilaiities that make it 
possible to suggest a common s^ioach to assessing program outcomes at the bacca l a ur eate, 
or first postsecondary dqpee level. In reaching this conclusion, the authors have drawn 
upon on-going studies conducted by the Institute for Management of Higher Education of the 
Organiiation for Economic Cooperation and Development (OEGD) in Paris and upon 
presentations at the International Seminar on Assessing Quality in Higher Education, which 
was held at Cambridge University in July 1989. 

Some Differences Among W estern Systems of Higher Education 

Student ai cc< ! ^ s to a university education differs markedly on the two sides of the 
Atlantic. The first universities in America drew extensively upon models fiom the colonists* 
native countries in Europe and Great Britain. Thus only the most capable few entered higher 
education, and graduates were prepared primarily for the ministry, law, medicine, or 
teaching. With the passage of the Morrill Act in 1862 and subsequent establishment of land- 
grant institutions, however, access to higher education in the United States was broadened 
considerably, and the number of fields for which graduates might seek preparation began to 
multiply. Currently a wide variety of baccalaureate-granting institutions flouiishts in this 
country because there is a societal expectation that virtually every person ca^le of 
graduating from high school who wants to attend college will have the opportunity to do so. 

While actions taken in the 1969s to accommodate the burgeoning post-war college-age 
population make higher education more accessible in most Western countries, universities in 
Europe and Great Britain continue the medieval tradition of offering education for a 
population composed primarily of the intellectual elite. According to Riebd (1989), just 
over 20 percent of tiiose completing secondary schooling in tiie Federal Rqniblic of Germany 
pursue postsecondary studies. By the year 2000, the Thatcher government hopes to increase 
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ftom IS to 20 percent the proportion of secondary school graduates who continue their 
education at a British higher education institution (Chamier, 1989). In a number of these 
Western nations, children are selected during their early yean of secondary schooling as 
potential candidates for higher education. They are placed in special preparatory curricula 
and then must pass rigorous written and oral school-leaving examinations in order to qualify 
for entrance to a university (Ottobre, 1979). Hiese students are not only intelligent, but also 
are highly motivated to further their education. In addition, they are likely to be better 
prepared than graduates of secondary schools in the United States, having spent more time in 
school (Porter, 1986; Schuler, 1984), and having taken a broad range of courses taught in a 
rigorous manner. 

In addition to differences in access that have a direct impact upon the character of the 
student population served in Western countries, there are also (Terences in the nature of 
validation processes for higher education in America and abroad. In the last quarter of the 
19th century, increasing competition for students among American universities led to such 
diversity among curricula that high schools were having difficulty preparing students to meet 
the vast array of university entrance requirements. The need for standardization led to the 
formation of first regional, then national, associations that assumed responsibility for 
accrediting institutions of higher education (Ned, 198Q. At present there are six regional 
accrediting associations in the U.S. and over 50 specialized organizations that accredit 
programs in fields such as business and nursing. Accreditation in the U.S. has evolved as a 
private, voluntary process, with institutions in a given r^on or specialists in a given 
discipline forming alliances for the purpose of setting standards and performing reviews to 
certify that the standards are being met (Rogers, 1989). 

In England and Europe, validation (accreditation) is a function most often performed 
by the government or its agendes. The Ck)uncil for National Academic Awards (CNAA) 
was established in 1964 for the purpose of assuring quality among British universities 
(Frazer, 1989). The CNAA approves courses of study, conducts annual monitoring 
activities, and provides institutional program reviews at five-year intervals. In France, the 
Ministry of Education's General Inspectorate moniton quality in that nation's institutions 
(Staropoli, 1989); and in the Federal Rqpublic of Germany, where 98 percent of the funding 
for universities comes from the state, institutions are established and accredited by the 
government. 

Universal Concern About Accountabilitv 

Europeans have long viewed competition— for reputation, outstanding faculty, c^nble 
students, and funding— as a hallmark of higher education in America. In the 1980s, however, 
competition was z world-wide phenomenon. With population increasing, many kinds of 
natural resources dwindling, and technological advances suggesting an ever-greater variety of 
enterprises in which monetary resources may be invested, every significant societal 
undertaking is being scrutinized in an unprecedented way to determine its efficiency and 
quality. Higher education has not escaped this scrutiny, despite its several-hundred-year-old 
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laging its own affairs. 



, the 1980s, accountability was the watchword for Western institutions of higher 
educ re universities careful stewards of the funds, both public and private, that they 

rece: t these institutions efficiently administered? Does the education they provide 
actuaii ^ value to the lives of their students? Are they fulfilling their mission of 
advancing knowledge in ways that bring tangible benefit to the societies that support them? 
These questions are being asked throughout Europe, Great Britain, and the United States. 

In the U.S., governors, legislatures, state coordinating boards, university trustees, and 
even the voluntary regional accrediting associations have begun to insist that postsecondary 
institutions demonstrate their accountability for the funding they receive by providing 
evidence that they are producing desirable outcomes. In England, where universities did not 
experience the intense emp^-'. -'s on practical education that followed the U.S. Civil War, 
there now is growing con / bout the contribution of higher education to economic 
development. The gover : ; wants to increase the proportion of youth who go to collie 
(Chamier, 1989), and the Department of Employment has launched a bold new initiative 
called "Enterprise Ijiitiative" to ensure that g^uates of British Universities will be well 
prepared to assume positions of leadership in business and industry (Elton, 1989). In 
addition, the University Grants Commission, which provides British universities with the 
bulk of their funds, has recently asked universities to introduce performance indicators and to 
provide evaluations of their work. They are also about to introduce a system of funding by 
contracts instead of through deficiency grants over a relatively long period (Kogan, 1989). 

In Holland, a 1985 Ministerial bill, "Higher Education: Anatomy and Quality," 
encouraged academic institutions to underteke their own internal evaluations, but also 
provided for evaluations by independent external visitation committees that operate under the 
auspices of the Inspection Office of the Ministry of Education (Drenth, van Os, and Bemet, 
1986). The National Board of Universities and Colleges, which is directly responsible to the 
Swedish cabinet, has been commissioned by that government to evaluate higher education for 
the primary purpose of gathering information for central decision-making (Furumark, 1981). 
In 1985 the President of France created a National Committee for Evaluation to undertake 
special studies designed to provide broad evaluations of French universities (Staropoli, 1986). 
The Committee reports directly to the President and is free to make public the results of its 
investigations. 

Peer Review: The Comm on Fvaluation Tool 

World-w me common responsibilities for preserving humankind's 

intellectual heritage ^ it to the current generation of young adults, for 

developing in these young people the powers to reason and to communicate their ideas to 
others, and for advancing the status of knowledge in the disciplines through individual 
scholarship and research. There are many ways to fulfill these responsibilities, and the 
greatest strength of universities m free societies is that professon and students have the right 
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to use their imaginations to create unique paths to their goals. 

Given the enormous diversity of acceptable goals for higher education, and the even 
greater variety of means for attaining those goals, it certainly is not surprising that 
academicians have not agreed upon a universally-applicable set of standards, performance 
indicators, or assessment strategies for gauging the quality of higher education outcomes. 
However, there is a time-honored evaluative process that is almost universally accepted 
within academe: Peer review- 

The voluntary accrediting associations in the United States have always relied upon 
the judgment of a visiting committee to ascertain the status of institutional practice vis-a-vis 
stated goals. In recent years the Dutch have begun to develop a similar s^roach to 
institutional evaluation (Vroeijenstijn, 1989). Since their founding in the Middle Ages, 
universities in England (Hton, 1989) have invited external examiners to zssi^ student 
achievement and thus maintain standards across institutions within the various academic 
disciplines* 

The authors have argued previously that the assessment of outcomes in higher 
education should be linked to on-going institutional processes such as strategic planning and 
peer review (Banta, Fishei; and Minkel, 1986; Banta and Fisher, 1989). The key point in 
this assertion is that these should be institutional processes, owned by Uie individu;il 
university, not externally-imposed processes. Only if each institution has the freedom to 
pursue its own goals using its own ^roaches and its own means of evaluation will we be 
able to nurture the creativity that can truly advance the status of knowledge* 

This is not to say tliat there should be no common methods for assessing quality. As 
indicated above, peer review, whether for the purpose of judging institutional quality or 
assessing individual student achievement, is one evaluative prociess that virtually ail' 
institutions accept. And since the judgment of reviewers must bt informed by evidence, a 
variety of assessment procedures, some of them common to gnyups of institutions, may be 
incorporated in the review process as sources of evidence. 

Student progress through the curriculum, including passi\ge of course sequences and 
intermediate examinations and ultimately completion of the course of study, is one indicator 
of program quality. As Drenthe and his colleagues (1986) at tlie Free University of 
Amsterdam have pointed out, student progress may differ accqjtably from one curriculum to 
another and among types of institutions. But faculty at each institution can establish their 
own expectations, then gauge student success against those standards. 

Student placement in employment, especially in jobs related to the fields in which 
they concentrated their studies, is another kind of evid&iice that faculty may wish to collect as 
they assess their programs. In many fields, placement of some students in graduate or 
professional education is an additional performance indicator. Program reputations certainly 
are enhanced by special recognition (honors and rewards) accorded the achievements of 
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current st acuity, and graduates. 

Pe IS of program quality as elicited in questionnaires and interviews constitute 
another impoi at genre of evaluative data. Students can be asked during and aAer their 
courses if thev nave learned content and skills at rates that are personally satisfying. They 
also can provide ratings of services, such as advising and placement, that have been designed 
to assist them in obtaining maximum benefit from their studies. Former students who did 
not complete courses of study should be asked why they decided to leave a program or the 
university itself. Graduates and their employers should be surveyed periodically to assess 
levels of satisfaction with career preparation provided by the universit}'. Students' parents 
may have valuable insights to share. Moreover, systematic efforts should be made to assess 
faculty opinion about the quality of student progress, institutional services, and program 
administration. 

While all of the indicators described above yield important information about 
institutional quality that can serve to inform the judgment of program reviewo^, perfas^s no 
odier criterion attracts more attention than that of accessing the competence of graduates. 

Assessing the Competence of Baccalaureate Recipients 

The earliest comprehensive examinations for baccalaureate candidates at American 
universities were similar, at least in form and purpose, to those given in Great Britain and 
Europe (Dresscl et al., 1961). That is, written and oral exams, the latter often administered 
by examiners from out'. Institution, were used to assess the extent of learning achieved 
by the carefully selecte.: ho had successfully completed a prescribed course of 

study (Smallwood, 1935). 

The earliest comprehensive exams were designed !?)ecifically for use in a given 
institution and for the purpose of certifying that individual students had achieved a certain 
level of performance pressel & Delisle, 1969). Later, faculty from several institutions 
began to collaborate on the baccalaureate exams, and now some of these are nationally 
standardized. In addition, in the United States, the focus on individual student achievement 
has shifted. Comprehensive exams axe being used more and more firequentiy to provide 
evidence of the qualiQr of educational programs as opposed to the level of individual student 
attainment. 

As long as curricula were virtually identical for all candidates for the baccalaureate, 
the common written exam and oral defense seemed quite zpptopnm. However, in die 
1890s, following the introduction of the elective system, use of comprehensive exams as 
criteria for awarding the baccalaureate in die United States experienced a decline (Jcmes, 
1933). During the first half of the 20th century, there v^as a resurgence of interest in 
comprehensive tesdng to certify individual attainment, but the student activism of the 1960s 
contributed to a sharp decline in this kind of testing. As colleges yielded to students the 
right to choose their own paths through die curriculum, they also gave up the responsibility 
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for certifying what students had learned. 



Among other societal influences, the call for accountability in higher education and 
other agencies supported by public funds produced in the 1980s a renewed interest in finding 
out what students know when they complete their baccalaureate studies. However, in the 
United States, as stated previously, the focus is more upon discovering through measurement 
of student achievement the strengths and weaknesses of curricula and methods of instruction 
dian upon settiing common standards of attainment and passing or failing individual students 
on the basis of tiieir exam performance. Even scores on exams administered by external 
professional agencies for purposes of granting licensure or certification to individuals also are 
being used by some institutions for program evaluation. 

In addition to tfie more traditional p^r-and-pencil instruments, U.S. faculties have 
begun to use performance measures that simulate experiences students will encounter after 
college. Supervisors' ratings of an internship or field experience, an in-basket exercise, a 
poster presentation, an interview, a series of speeches, and a portfolio, are examples of such 
measures. In some of these performance samples, notably the videotsq)e of speeches made at 
various times during the student's career and die portfolio of writing samples, feculty interest 
is not so much in the relative levels of students' final performances as in the amount of 
growtii each exhibits throughout the series of behavior samples. While the graduates of a 
given institution may not be capable of functioning above the national average on a 
standardized exam, their college experience may have contributed substantially to their 
development of important skUls. 

In the United Kingdom and Western Europe, candidates for tiie baccalaureate are 
presented with broad topics that call for understanding of concq>ts in their fields of study, 
tiien are asked to write essays that demonstrate tiie breadth and depth of tiieir understanding. 
A diesis or report on an original project may also be required (Porter, 1986). Subsequently, 
these students must be prepaied to provide on oral defense of tiieir written work if required 
to do so. Panels of reviewers tiien pronounce judgment as to the fitness of each candidate 
for graduation and the class of degree to be awarded. Students who do not pass their 
examinations do not receive degrees. 

A few selective institutions in tiie United States also employ comprehensive 
examinations to certify die competence of graduates. However, no more than half of tiie 
postsecondary institutions in tiie United States have a comprehensive examination 
requirement (El-Khawas, 1989), and most of tiie remainder employ such measures primarily 
for program evaluation puiposes. 

Discussion 

Substantial differences still exist on tiie two sides of the Atlantic between tiie levels of 
preparation, ability, and age of students entering postsecondary institutions. Moreover, 
differences in sources of validation and support for institutions have an infiuence on tiie 
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^^^^ 1 assessing baccalaureate program outcomes. It might appear, for instance, 

that ui js in the centrally-planned systems of higher education in Great Britain and 

Europe be subject to a more uniform set of evaluative criteria than the universities in 

Amwic iTt accreditation has traditionally been conferred by voluntary associations of 
institutic 



Howevtt, all Western universities have maintained a substantial degree of 
independence from external influences, and thus are more alike than different in that they are 
niore attentive to standards they have set for themselves than to external criteria. In 
addition, such factors as improved conununications technology, increased world-wide 
competition for resources, and demands for accountability have served to increase the 
similarities and mitigate the differences among systems of higher educaticii in the Western 
world. For instance, since tht 1960s, American universities have come under increasing 
state control (Kells, 1986); and in Europe, universities are experimenting with voluntary 
associations to assess and strengtiien program quality (Vrodjenstijn, 1989). 

Peer review appears to be an almost universally accepted evaluative process in 
Western universities. The authors believe that outcomes assessment activities can, quite 
naturally, be incorporated in program reviews as a means of providing evidence to inform 
tiie judgment of faculty as well as external reviewers. Maximum effectiveness will be 
realized if peer review is an internal process, initiated by the individual university, focused 
upon its goals, and aimed at improvement as opposed to mere assessment of status. 

If Western universities can convince governmental autiiorities that internally-initiated 
peer review is die healthiest form of evaluation for higher education, then we should come 
togeUier to stiidy the process and perfect it. Scholan on both sides of tiie Atlantic have 
already begun to contribute to that dialogue. 

Elton (1989> h?.' w«»(^•'<:^?:d that external reviewers receive some training for Uieir 
work. Studies cor ^5) of British external examiners iweal that while 

these assessors of stuceiii , . are eminent persons chosen for their knowledge of a 

field and experience in it, they are largely uninformed about good practice in designing 
examinations, know littie about statistics, and base their decisions primarily on tiicir own 
experience rather tiian upon theory or systematically-collected evidence. 

Findlay and AUsop (1989) are strongly committed to internally-initiated program 
review, but recognize that external autiiorities need simply-stated evaluative summaries and 
tiius often resort to tiie use of quantitative indicators. These investigators are using document 
analysis, observation, and interviews to identify tiie evaluative criteria that academics at 
Portsmouth Univenity in England use in making judgments about program quality. They 
hope to develop ways to supplement quantitative data about programs witii easily readable 
summaries of qualitative judgments. 

In considering cross-national comparisons of stiident achievement, Harris (1989), tiie 
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U.S. representative on the OECD commission currently investigating this topic, has 
suggested that a useful beginning point would be to compare the time allotted in the various 
univwrsity systems for students to study a given discipline, or major constructs or topics of 
the discipline. This is a basic step in curriculum mapping, which is another technique that 
could be useful in program review. 

Finally, Lapointe (1989), who directs the National Assessment of Educational 
Progress for elementary and secondary school levels in the U.S., has proposed a "second 
generation measurement program" with potential q)plicability in postsecondary institutions. 
This program would include (1) a community profile that characterizes the larger 
environment in which education takes place, (2) a description of the environment for learning 
in the unit being reviewed, (3) surveys for students and instructors that elicit descriptions of 
classroom practices and teaching strategies, (4) a set of student performance tasks designed to 
measure higher-order intellectual skills, and (5) a set of achievement measures in a variety of 
disciplines. 

We have much to learn from each other, and collaboration is becoming increasingly 
important. Universities must clearly establish their right to independence; but in order to 
safeguard this right, each must chart its course explicitly and prepare itself to furnish credible 
evidence of the quality of its performance. 
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Institutional Research and Planning to Support 
Decentralization and PrivatiTatinn! 



Building Self-Correction and Accreditation Systems 

Robert C. Fioh 
Syracuse University 



The Problem 

Higher education in Eastern Europe fiaces sr'eial significant problems during the 
democratization of government and education. Disparate institutions have become 
autonomous in a management sense, but with Inaricqnafc mechanisms for accountsd>ility and 
quality control. In addition, they remain almost totally dqiendent on a declining base of 
financial support fiom diffeient govonments and dqKutments. Demograq)hic trends coupled 
with the fact that only a small percentage of the eligible student population currently enrolls 
Mi-time at universities and collies suggests that pressures will continue to build to inoease 
the number of spaces available to students. However, the employment market )ias declined 
precipitously and many who graduate will have great difficulty in finding worl^ While all 
these pressures suggest a need to make many changes, the disparate and traditionally 
conservative nature of higher education institutions prevents them fiom addressing these 
problems. New management and planning systems are needed. 

Some of the major transition issues at this point are: (1) building better self 
monitoring and improvement mechanisms, (2) affirming a collaborative accountability to the 
audiences and markets of higher education, and (3) exploring new markets such as business 
and education consortiums to insure more financial support, employment for graduates. The 
evolving audiences and markets of these institutions of incoming students, research support 
dollars, employees of graduates, alumni, and government agencies must be better understood 
to reduce the potential of externally imposed control. 

Potential Contributions of Institutional Research in the United States 

Institutional research/planning (JSP) processes and organizational structures like those 
that have matured in higher education institutions in the U.S. could provide models to 
support the transition of higher education in Eastern Europe fiom central state control to 
decentralized management. 
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Increasingly, institutions in the United States see the need to collaborate with each 
other to support IHP that addresses changing students and research markets. Declines in 
traditional student populations and resources for sponsored research have created a climate 
that demands both competition and collaboration. IRP models that support the management 
of institutions and the planning of consortia of institutions would be particularly helpful to 
Central and Eastern European universities looking for opportunities to work with other 
institutions in addressing common educational issues. Private institutions such as the Higher 
Education Data Sharing Consortium (HEDS) of which Syracuse is a member, provide case 
studies of balancing competitive and collaborative forces to strengtiien the quality of 
academic programs. Academic programs within private institutions enjoy on average more 
autonomy than tiiose within public instituticms. 

Three key methods used in IRP would be demonstrated to show higher education 
leaders in Eastern Europe a paUi towards both decentralization and increased quality. The 
use of these tiiree key metiiods within a decentralized management structure at various 
institutions would build essential processes for self-study and external reviewers would build 
essential monitoring systems for relatively autonomous institutions. 

Implementing Institutional Research and Planning in E astern Eumpe 

Developing stronger IRP can be accomplished in the following ways: first, through 
visits to selected Eastern European institutions by researchers/planners to demonstiiate IRP 
piDcesses with senior £unilty and administrators and to help set up IRP organizational support 
structures; and second, tiirough visits to U.S. institutions by selected staff for periods of one 
semester (15 weeks) to observe and participate in IRP where academic lead&rs have 
significant administrative autonomy. 

Initial contacts would require establishing a frame of reference regarding the 
leadership styles, organizational structures, and values held by senior faculty and 
administrators at selected institutions in Eastern Europe. Initial contacts would also establish 
ongoing working relationships with faculty and administrators. 

Visiton to selected Eastern European institutions should recognize the various levels 
of higher education tiiat serve somewhat distinctive purposes: first, ^rmiM of research 
and universities of arts and sciences; second, universities of economics, universities of 
medical, technical and agricultural sciences, and colleges of arts and physical education; 
third, colleges for teacher training, tedmical training, economics, elementary sdiool teacher 
training, agriculture, medicine, and state administration. It is essential to identify the major 
issues faced by these various levels of institutions such as establishing stronger links between 
research academies and first-tier universities, strengthening the business market place for 
second-tier institutions, and building stronger higher education and secondary education 
collaboration for third-tier institutions. 
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Developing the Tools for Using National and Tntematf^nal 
Data Bases: The Issues of 
Content. Assessment, and Application 

E. Stephen Hunt' 
U.S. Dq)aitment of Education 



Introduction 

Comparative postsecondaiy education research may be divided into two categories: 
theoretical research and applied research. Theoretical research focuses on applying social 
scientific methods and principles to the study of postsecondaiy education in different 
societies, including the disciplines of sociology, economics, politics and-most especially- 
anthropology. Indeed, the true spiritual home of most academic compaiaiivists would appear 
to be the cross-cultural perspective of ethnognq)hy, together with related research methods. 
Certainly most U.S. university programs in comparative education tend to be located in the 
social foundations divisions of schools of education or in anthropology or sociology 
departments. 

Applied comparative research consists of all the activities undertaken by educational 
institutions, government agencies, foundations, and educational associations to develop and 
apply standards for relating the educational processes and products of one system to those of 
another. This includes such activities as developing commcm terms and operational 
definitions, developing and maintaining data bases, analyzing transcripts and other evidence 
of curriculum content and academic achievement, interpreting differing regulations and 
management systems, and other activities that in the United States are the province of deans, 
registrars, institutional research officers, association and foundation staffs, and public 
officials. Applied comparativists (who may not see themselves as such but who cctiduct 
comparative research all the time) tend to come from a wider range of backgrounds than 
academic comparativists, a situation that enriches dieir p erspective on issues even as it 
sometimes restricts their in-dq>th understanding of the cultures wi& which they must deal. 



'This paper was prepared by E. Stephen Hunt in his private capacity. The views are those 
of the author and no official support by the U.S. Department of Education is intended or ^ould 
be inferred. 
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-'niributions of both groups of compaiativists are critical to achieving progress 
towai velopment and use of practical and sensible methods of measuring educational 
achie^ and quality across national boundaries (and even within some countries), 
Acadi' omparativists have helped those charged with this task to avoid crude "box score" 
compa. IS that are not based on an understanding of underlying issues or structural 
differer.-es, and they have reduced misunderstandings due to misreading of evidence or 
confusing forms with substance. But they have occasionally criticized the very nature of the 
task itself, ^jparently because the anthropological research ethos of so many comparativists 
recoils at the notion of direct comparative assessment and, most especially, the underlying 
implications of competition and judgment. Applied comparativists have not always 
recognized or appreciated this perspective; this has led to several failed attempts at 
cooperation. 

Things do not have to be this way, but overcoming differences in professional 
socialization is not easy, especially when the groups involved are as different as culture 
scholars, ^lied statisticians, and public policy professionals are. There are, however, quite 
enough objective difficulties to overcome in developing the sort of modalities proposed'at this 
conferwice to make differences of approach pale by comparison. I would cite but three of 
these difficulties: creating comparable data bases; creating crosswalking mechanisms for 
actually using such comparable data; and developing standards and definitions for concepts 
that permit the collecticm of data in such a manner as to allow the data bases and the 
crosswalks to function as intended. These will be discussed in reverse order. 

Standards and Definirinns 

Before comparison of any kind can proceed, concept definitions and standards for 
related data input must be accqnable and agreeable to all parties concerned (both the political 
and research interests) and must be shown to produce reliable results. Since sovereign states 
are rarely likely to surrender their own internal definitions and standards in favor of someone 
else's, and since many countries (such as the United States) operate without a uniform system 
of definitions and standards, comparison has to be made via a laborious process of pushing 
for cooperation where possible and making up for it where it isn't. 

Pushing for cooperation, of course, means active participation in international efforts 
to arrive at common data systems. Many of us in government are interested in having the 
United States actively partidpate~at the technical level as well as the poIitical~in efforts by 
the O.E.C.D., the European Community, and other country groups to develop such 
intemationaUy useable systems. Informal efforts are also welcome, as typified by the work 
of the ERIC Qearinghouse on ICgher Education in developing an international dau-sharing 
network. These efforts will, it is hoped, receive encouragement from current interest in 
using international ccmpaiisoos to spur reform in U.S. education. But in the meantime much 
can be done to improve comparability using eusting arrangements. 

One project currently underway in the Higher Education and Adult Learning Division, 
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where the author works (and under his lead), is the development of a crosswalk from the 
ISCED to our federal standard classification system, the CIP (Classification of In itructional 
Programs). The author was fortunate to be one of the co-authors of die 1990 CIP which is 
an extensively revised and updated edition of this system~in many ways virtually a new 
system. When finished, this crosswalk will provide users with a reliable method of 
interpreting U.S. instructional program data using ISCED, one that will also be officiaUy 
approved by the Department of Education and other Federal agencies. The develooment 
stage of the ISCED/CIP Crosswalk has begun; it should be ready for use sometime in late 
1992 or early 1993. someame m late 

Many other existing systems can be made compatible for comparative purposes by 
developmg mechanisms such as a comparative crosswalk of terms. A very useful task in 
this autiior's view, would be for comparativists to inventory such existing systems with a 
view to determining whether compatibility is possible, and if so, to aUocate responsibility for 
domg die work among tfiemselves. Partial data utiUty might be possible even ifsystems 
cannot be crosswalked directiy. This would certainly be better tiian nothing at all. 

Developing Data Systems 

Beyond crosswaDdngtaxonomic concepts lies die territory of coding systems and die 
deagn of mediods and systems for accomplishing comparisons. This can be highly technical 
as in the case of die design of computer software for manipulating data, or merely highly ' 
complex, as in die development of coding systems for ordering educational data ftom 
different sources in a logical and uniform way. The latter task is the subject of anodier 
project on which dus audior is woridng, and also of a very important ongoine oroiect in die 
pnvate sector. * *^ 

One of die most significant educational data bases of die United States Government 
called die Survey of Earned Doctorates, a survey of aU doctorate recipients graduating from 
U.S. institutions. It is conducted by die National Academy of Sciences for a consortium of 
federal agencies diat includes die Department of Education and die National Science 
Foundation, and has been u.idcrway since 1920. The importance of tfiis data base for 
comparative purposes is diat it contains background data on all foreign students who earn 
U.S. doctorates, which adds up to a small but significant sample population each year-and a 
growing one. The background data include mformation on country of birdi, place of birdi 
(dty and region), country of current citizenship, academic career since seccmdary school 
country of residence (sometimes different from diat of birth and citizenship) suWects studied 
at each level of education, and future plans. ITiis is a rich data base for all 'sorts of analyses 

but It requires die dcvdopment of an up-to-date coding system diat pennits die forden data ' 
to be used, mcluding a system for assigning variable codes to different institutions aiui 
degree levels across countries. 

-^e Higher Education and Adult Learning Division is cooperating widi die National 
Saencc Foundation m developing a master coding system for die foreign institutional data 
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collected Jivey of Earned Doctorates. This system will allow U.S. and foreign 

institution. datti to be compared on a comprehensive basis, and will establish a 

workable gy of institutions. When this stage is concluded, the project will expand to 
link the ISt CIP Crosswalk and the SED Coding System, thus developing a useful tool 
for compan. academic work by level, content, and institutional or d^ree type. The initial 
stage of the j^ED Coding System may be ready later this year. 

This project is indebted to another, much older project along the same lines but using 
more traditional m^ods, orld Education Project of the American Association of 
Collegiate R^strars and Admissicms Officers. For many years this private initiative has 
endeavored to produce accurate and practical guidelines for use by postsecondary institution 
officials in assigning credit for non-U.S. academic work, placing foreign students, and 
interpreting foreign credentials. It is a quiet but worthy effort at applied comp:>:.-ative 
research that deserves to be much better known than it is, and should receive more support 
for the continued production of guiddxmks. 



Conclusion 

Fundamental appUcd research of a descriptive and analytical nature is essential to 
developing effective measurement, assessment, and quality control procedures on a cross- 
national basis. Hie knowledge base of those who will implement such modalities needs to be 
increased, and this in turn will require better data bases and their essential prerequisite, 
better data input. If that sounds like too much suspicious jargon, try this: you can't get there 
from here without doing two things-broadening your understanding of "tfiere" and why you 
want to get to it, and improvirc; the tools you need in order to make the joum^. The 
Higher Education and Division of the U.S. Department of Education's Office 

of Research hopes to ii: isting the field in the devdopment of some better 

descriptive tools, as weti as by cooperating with others who are interested in improving the 
ability of the comparative education research community to serve both its own research needs 
and those of society. 
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New Armaches to Evaluation in Aust rian Higher Brinratinfi 



Hans Pechar 

Austrian Ministry for Higher Education and Research 

Introduction 

New s^yproaches in die evaluation of Austrian higher education have been the subject 
of recent discussions. These discussions are related to changes in higher education policy 
and a new assessment of priorities, problems, and challenges in Austrian universities. 
During the 1960s and 1970s, ejcpanding and democratizing higher education was the priority. 

In that time the steering function of the centndized governing body was strengthened 
and authorities as well as the public paid a great deal of attention to the system's overall 
performance. During the 1980s, the maintenance and improvement of quality became the 
most important issue. This chan^ was accompanied by a trend towards a decentralization of 
authority and attempts to strengthen institutional autonomy. More and more attention has 
been paid to the performance of (and the differences in the performance oQ single 
institutions. However, these new s^proaches in the evaluation of Austrian higher education- 
that is, those emphasizing the relative performance of individual institutions-face serious 
obstacles. 

Higher Education Policy in the 1960s and 197Qs! What Kind of Tnfnr mation about the System 
Was Thought to be Brievant? 

In the 1960s, when Austria began a deliberate expansion of its higher education 
system, a two-fold rationale emerged for doing so: 

—the expectation of a significantly growing demand for graduates; and 
—the need to democratize and broaden access to higher education. 

In the Austrian context of a highly selective secondary school, the most important 
prerequisite for meeting these goals was to expand and reform the "gynmasia" (the elite track 
of secondary schools). Another important stq) towards mass higher educatim was the 
raticMialization of the curricula in universities. The mixture of medieval customs and 
traditions that dominated Austrian higher education was to be clarified and replaced by a 
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system of unified legal regulations for courses of study. These regulations would bind all 
Austrian universities. This legal system would enable the authorities to make tiie universities 
responsive to the demands of society and the economy. 

These legal reforms resulted in an increased responsibility for the ministry, wii^h ; 
acted as a centralized governing body as opposed to the sdf-goveming bodies of the single 
institutions. In other words: an organizational pattern that already dominated Austrian higher 
education was remarkably intensified. Burton Oark refers to this pattern as the "continental 
mode of autiiority distribution."* Hus organizational pattern is characterized by the 
concentration of authority at the bottom (diair-holder) and at tiie top (ministry) and a weak 
autiiority at tiie enterprise level (the single institution), meaning a strongly centralized 
administration witii littie institutional autonomy. 

What is important in our context is tiie feet tiiat policy priorities as well as 
organizational characteristics shaped the evaluation of Austrian higher education. The 
dominant question centered on Uie success of the expansion and democratization of higher 
education. To deal witii that question tiie administration created a remarkable system of 
statistics tiiat provided inform ^ about the overall performance of Austrian higher 
education. A number :: (the most important is tiie "Rqmrt on Higher 

Education," which is publisii^ every tiiree years) make tius information available to tiie 
legislator and the general public. 

This type of information reflects tiie organizational patterns of Austrian higher 
education. It is provided by tiie central governing body responsible for tiie overall systems; 
tiierefore, every problem is addressed at tiie overall level. Since Austrian higher education is 
highly homogeneous and since each institution is supposed to have the same standards and 
characteristics, information about tiie institutional level (especially about differences between 
single institutions) was n^arded as less important. The single institution must conform with 
tiie central regulation and tiie wisdom of tiiese regulations is supposed to guarantee high 
quality. Under these circumstances there is no need for the single institution to undertake a 
self-study nor is tiiere any interest in tius type of information. 

Issues of die 1980s: Quality and Institutional Autonomv 

Beginning in tiie late 1970s new issues emerged and eventually rq)laced tiie former 
priorities in higher education policy during tiie 1980s. The growing number of students was 
no longer regarded as a desirable improvement of tiie labor force but rather as an ovenupply 
and a burden for univer^ines by endangering the quality of higher education. Those who 
stress the question of c refer primarily to tiie long duration jof studies and the high 



> Burton R. Clark, The Higher ducation Svst emr Academic Organization in Cross- 
National Perspective . Berkeley: University of California Press, 1983. 
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drop-out rate. 



Few students graduate within the nunimum time required for a degree and more than 
half take five or six semesters longer than the minimum. This fact implies a rather advanced 
age of graduates and a late entrance into the labor market. The number of students who 
complete their courses has continuously declined in the past and the drop-out rate is now 
above 50 percent. Thus expansion of Austrian higher education is primarily one of the 
students, much less of graduates: whereas during the last two decades total enrollment has 
more than quadrupled, graduates have hardly doubled. 

Although those problems arc not entirely new, they preceded the expansion of higher 
education and now attract more attention because 

—they have worsened in relative terms (increasing average length of studies, 
increasing drop out rate); 

—they have worsened in absolute numbers due to the increase of students, 

—in die context of the financial difficulties of universities, high drop-out rates are 

considered a waste of money. 

There is a growing consensus among esaperts as well as the public that it is necessary to 
increase the "throughput" in Austrian higher education. 

At the same time there are serious discussions about the distribution of authority in 
Austrian higher education. Many experts hold that the weak institutional autonomy of 
Austrian universities is at least one reason for the problems mentioned above. Tlie 
conviction of the 1960s that a strong central govexning body is more appropriate to the 
transition to a mass higher education system is r^o longer shared by the majority. 
The central administration had to realize that t!ie real power to steer and even influence the 
system is much smaller than its l^al authciity. 

New Approaches to Evaluation in Higher Education 

Again, the changes during the 1980s had remarkable consequences for the evaluation 
of higher education. The information provided by official reports is absolutely insufficient to 
deal with the problems of drop outs or the long duration of studies. To give only one 
example: The "Rqwrt on Higher Education" contains an impressive number of statistics 
about new entiants and total enrollment broken down by any number of students who make 
serious attempts to complete their studies (as opposed to "Fictional Enrollments," that is, 
students who are only enrolled because th^ receive some benefits; guesstimates for those 
students range between 5% and 25%). There is even less information about the educational 
process of the enrolled students. Obviously the central administration has no access to this 
type of information. Universities, on the other hand, have littie interest in this type of 
information. As mentioned they have neither an obligation nor an incentive to undertake a 
self-study that could answer at least some of tbt relevant questions. 
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However, the new issues of quality and institutional autonomy have increased the 
interest in infonnation at the institutional level. On the one hand only infonnation at that 
level is appropriate to the uncovering of educational deficiencies. On the other hand, this 
information is crucial to the implementation of new types of central administrations that are 
compatible with increased institutional lutonomy-that is, administrations which confront 
institutions with a pattern of incentives instead of detailed prescriptions. While this type of 
steering enhances the discretion of the institutions, it also requires more accurate information 
about the institutional performance. 

Up until now the central ^administration has come forward with a mixture of 
encouragement and regulation to stimulate universities to undertake self-evaluation. Most 
remarkable in this context is an amendment to the University Organization Act that was 
passed last year which obliges the Minister for Science and Research to guarantee a periodic 
evaluation of the single institutions. 

There are some promising beginnings in that process, such as the evaluation of 
physics research in Austria. This procedure, which is organized by the Austrian Physics 
Society, involves the entire community of physicists in Austria. Its final teport will provide 
a basis for decisions about future developments. 

However, there are also serious obstacles to the advancement of institutional 
evaluation, especially a «autual lack of trust between the ministry and the universities. 
From the point of view of the ministry the institutions have to prove thdr ability to manage a 
higher depee of sdf-govemment One way to prove this is to undertake serious self-studies. 
The universities are mistrusir. ' as well. What use would the ministry make of the results of 
such self-evaluation? This, in /act, is nc^: dear even within the ministry. The rqports may 
have no oxisequence at all, which discourages those in the university community who 
support self-evaluation. Therefore, it is important to de^'dop a procedure that will reduce 
this mutual lack of trust 



80 

^ BEST COPY AVAILABLE 

ERIC 



Assessing the Quality of Higher Friiif.arinn- 
A Mexican Persoertivft 



Felipe Martinez Rizo 
Universidad Autonoma de Aguascalientes 
Aguascalientes, Mexico 



The Mexican Higher F/<iiratlon System: An Overview 

Twenty years ago, in 1970, the Mexican ffigher Education System (MHES) bad 
271,275 students. In 1989, the system had an enrollment of more than 1,200,000 students. 
This includes the Normal Schools (where elementary school teachers are educated) which 
have had university-level status since 1985. 

As a result of demogr^hic growth and of increases in the proportion of cohorts that 
have access to the university level, the expansion of the MHES has been impressive. 

The system includes different kinds of institutions: public (state-supported) and 
private; large universities, medium-sized technological institutes, other smaller professional 
schools and technological institutes, other smaller professional schools and now, normal 
schools. Since the latter also serve as in-service training centers for students who wish to be 
school teachers, not the same kind of students as those in the other Higher Education 
Institutions (HEIs), the normal schools will not be considered in the following analysis. In 
fact, to including normal schools in the analysis would be misleading, since there are more 
of these than other higher education institutes (346,225 public and 121 private), but they only 
have 131,046 students. 

The other, more conventional higher education institutes total 325, distributed as 
follows: 
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TABLE 1 



The Mexican Higher Education System 
Undergraduate Students, 1988-89 



Type of Listitution 


LastitutioDS 


atudents 




PUBLIC INSTITUTIONS 


155 


898,420 


5,796 


Universities 


40 


731,592 


18,290 


Technolosical institutes 


94 


159,239 


1,694 


Other professional schools 


21 


7,589 


361 


PRIVATE INSTITUTIONS 


170 


171,145 


1,007 


Universities 8c Technological 


28 


114,941 


4,105 


institutes 








Other professional schools 


142 


56,204 


396 


TOTAL 


325 


1,069,565 


3,291 



Source: Anuaho Estadistico 19S9. Licendatura ANUIES. 



Considering that they are institutions of very different size with the same institutional designation (very 
large and very small universities, schools, etc.), it is interesting to see a classification of die HEI's 
according to their enrollment figures: 

TABLE 2 



Mexican Higher Education Instibitions by Size 



Students 


PliblkHETs 


Private HETs 


Total 


More tiian 100,000 


1 




1 


50,000 to 99,999 


4 




4 


25,000 to 49,999 


4 




4 


10,000 to 24,999 


8 


4 


12 


5,000 to 9,999 


13 


3 


16 


3,000 to 4,999 


9 


8 


17 


Less tiian 3,000 


116 


155 


271 


1 TOTAL 


155 


170 


325 
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As is easily seen in Table 2, 17 public higher education institutes and only 4 private 
institutions have 10,000 students or more; 138 public higher education institutes and 166 
private have less than 10,000 students. 

If the mean for students in public universities is 18,290 (Table 1) it is easy to see that 
these institutions are large. This include the huge National University of Mexico City 
(UNAM, with 134,150 undergraduate students). At the same time, all other higher education 
institutes have (with a few exceptions) less than 3,000 students. 

The Quality of Higher Rriiicadon Institute in Mexico: A First Appmarh 

With very few cxcq)tions, the 155 small, private higher educsiticn institutions do not 
have good levels of quality, as £tf as it is possible to jud je from their resources: they have 
no full-time faculty; their libraries are very poor, and they have no laboratories. Obviously 
they have no research activities and, in many cases, their management is concerned with 
profit and not with academic purposes. 

The inability of the inexpensive (or free) public higher education institutions to cope 
with the huge serial demand for higher education in the 1970s caused the private education 
institutions to flourish. The lack of real control from the ^Gnistry of Education (Secretaria 
de Educacion Publica, SEP), in spite of its legal liabilities, contributed to this growth. 

On the other hand, the larger private HEIs are quality institutions with well-fiunished 
and equipped campuses, good libraries, M-time Acuity members, and research activities. 
They attrzct their students from the small affluent secton of the population that can affoid 
their expensive fees, and receive economic support from large corporations. They are also 
well staffed and efRdently managed, without the union problems of public institutions. 

These large private higher education institutions are also impressive from an outcomes 
perspective. Their alumni are well r^arded for tbar qualifications; in fyct the leading 
private institutions, such as the Technological Institute of Mbnteney, the Univ. 
Iberoamericana, and the Autonomus Univ. of Guadeiajara are unanimously considered as part 
of the better higher education institutions in Mexico. 

The Public Tnstitiirinn« 

The 21 smaU "other professional schools" are very heterogeneous and include, for 
example, the Army's officers school, and some fine arts and music schools. We will not 
include them in our analysis. 

The 94 public technological institutes are, by contrut, very homogeneous: they are 
sniall, well-equipped, with a good number of full-time faculty memben, but with some 
important problrais that prevent £»nilty members from offering good qinlity teaching. These 
include very poor operating budgets which leave large laboratories inoperable and library 
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buildings wi it books; large union problems which produce an incredible faculty 
absenteeism Jt; very highly centralized academic planning with curricula and teaching 
programs den ied in Mexico City Oeaving faculty largely unaware if not totally alienated 
from this process); and a very high level of political unrest among students which leads to 
frequent strikes and campus conflicts. 

The 40 public universities are, once again, very heterogeneous both within and among 
themselves. In fact many local campuses or schools are so large (with 10,000 students or 
more in each) that these differences are logical. Some of these units arc indeed very good, 
but the general level is quite deficient. 

Some of the reasons for the deficiencies are the same as found for the previously 
considered non-autonomous, technological institutes-poor operating budgets (especially from 
1981 on)--but other reasons are different. As autonomous institutions, public universities 
have their own unions, and in many cases a very strong power struggle leads to confrontation 
between right and left radicals and the official party. In fact most higher education 
institutions are ruled more or less openly by one or another of the political parties. 

It is important to note that some public high quality graduate and research institutions 
that are not being considered. In some instances undergraduate education is not as good as 
graduate education, as seems to be the case for the UNAM, which has 9,617 graduate 
students, not to mention more than 100,000 senior high school students, for a total of nearly 
300,000. 

The Need for a More Pre d w Fvaln atinn- Vt^ ^t Merican Efforts 

The fu:st attempts to evaluate quality in a Mexican higher education institution 
occurred in 1967, when the Technological Institute of Monterr^ started a self-study process 
with the methodology and advice from the American Southern Association of Colleges and 
Schools. Since that time the Institute has been reviewed by the American Southern 
Association of Colleges and Schools. The Institute is the only Mexican higher education 
institution that, as an institutioa, is accredited in the U.S. Some programs of other private 
institutions have also submitted themselves to q)ecialized accreditatioa. 

In 1969-1970 four other higher education institutes (two public and two private) also 
conducted self-studies under the advice of Dr. Pablo Latapi from the then recoit ly 
estabUshed Center of Education Research (Centro de Estudios Educativos, CEE), 

This center prom. of the self-study methodology, and the Ministry of 

Education, SEP, started a pmgraiu m 1974 to make public universities conduct self-studies. 

By that time other private higher education institutes had conducted their own self- 
study processes and started creating planning and institutional research ofGces, as was also 
the case for the Universidad Autonoma de Aguascalientes. Established in 1973, it is one of 
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the few public HEIs where those efforts s u cceeded. In most cases, planning or institutional 
research offices were established at public institutions because of the requirements of the 
Ministry of Education, but were not operational. Some other excq;)tions were the public 
universities of Nuevo Leon, Yucatan and the Metropolitan (established in 1974). 

Beginning in 1978, the Ministry of Education, together with the National Association 
of Universities (Asociacion Nadonal de Universidades e Institudones de Educadon Superior) 
SEP-ANUIES proved a national "Plan* for Higher Education. In 1979 a permanent- 
planning system was established, which considered the national and regional levels. It is 
based on institutional planning offices or "unidades institudooales de planeadon." 

At its 1983 meeting, the National Association of Universities (ANUIES) tried to 
establish a common system of evaluation; as the proposals received firom different higher 
education institutions at the meeting were difficult to int^rate, the Association decided to 
hold another special meeting in 1984, to consider an evaluation system. 

For the 1984 meeting, staff prepared a document that, in fact, the Association 
approved as the official guidelines for evaluation, and lecommendfid that the Mexican Higher 
Education Institute voluntarily adopt the model. 

In spite of those recommendations, each institution continued doing as before, and the 
economic crisis that struck the country fiom 1982 on and especially in 1986-87 produced 
strict budgetary cuts and unrest becauss of wage reduction in real terms, forcing these issues 
to become the most important concern. 

After the 1988 elections, the federal government launched an ambitious six-year 
program to recover economic growth. The success in the nsnegotiaticm of external dd>t in 
1989 permitted the government to promote its modemizatim program. 

At the higb^ education level, the program includes as an important focus the theme 
of tht evaluation of higher education institutions. 

The "Program for the Modernization of Education" establishes a holistic evaluation 
system which includes five aspects: student performance Onduding botii cognitive and non- 
cognitive elements); curriculum and teaching; educational management; public educational 
policy; and die social impact of education. 

The evaluation is conceived as a participative enterprise, involving all persons who 
take part in the educational process who have to be, at the same time, agents and objects of 
the evaluation process. 

Specifically for higher education, the program describes the task that must be 
accomplished: 
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The r.otion of a national process for the e\'aluation of the Higher Education 
■ysiem for assessing its present level of poformance, 
productivity, efficiency and quality . . . The Process will be 
conducted technically by a National Commission . . . that will 
behave with the consent of the HETS ... and will be based on 
a set of guidelines that have to include the participation of the 
institutions themselves ... (p. 141). 

In November 1989 the National O)mmission for the Evaluation of Higher Education 
was in fact established, with a sprdalirrd staff that in May 1990 produced a new document 
with criteria and guiddines for the evaluation that was to be started that same year. 

The Experience of the Autonomous University of Aguascalientes 

Situated in the center of the country, 500 kilometers north of Mexico City, the State 
of Aguascalientes is one of * aller in the Mexican Republic, with 5,471 square 
kilometers and an estimate. voiation in 1990 of 750,000. More than 500,000 are 
concentrated in the capital cay of the state, w^ose name is also Aguascalientes. 

Established in June 1973, the Universidad Autonoma de Aguascalientes (UAA) is one 
of the youngest public autonomous higher education institutions in the country. New 
institutions of this kind have not been established since 1974, and the tremendous growth of 
the Mexican higher education institutes has been largely supported by the same 40 
autonomous universities (whidi have become very large, as we have seen), and by many 
smaller non-autonomous technological institutes and private higher education institutes. 

In 1990 the UAA has 5,200 students at the licendatura level (roughly correspondent 
to the American baccalaureate i£vd, but with a professional training emphasis: in hct aU the 
liberal professions, as jrihysicians, lawyers, architects, etc. are formed at the licendatura level 
and do not require a master's or a Ph.D. d^ree). 

These 5,200 students are distributed in 32 professional fields ftom medicine and 
dentistry to dvfl engineering and architecture; from law, business administraticm and 
accountancy to sodology, history and Spanish literature; firom agriculture and veterinary 
medicine to biology, biochemistry and mathematics. 

In addition to the licendatuia-levd studies, the UAA offers technical levd courses in 
nursing and industrial design to some 500 students, and high school studies to other 1,500 
for a high total of mor iian 7,200. 

Sixty-five percent o. tion's annual budget is d^oted to teaching in these 
different levds, while 8 percc ated to research (mainly in basic sciences, social 

sdences and education, and agn. 7 percent goes to service activities related to die 
community of Aguascalientes, ana cent is spent in direction, management, and sui^port 
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The institution has acquired a solid rqnitation as one of the country's leading public 
higher education institutions, with an outstanding record specifically in planning and 
institutional research. In 1976 the UAA started its first self-study and institutional planning 
process, which produced its first "Plan de Desanollo" (Development Plan) for the yean 
1977-1983. In 1982 the second Plan de Desanollo was prqiared, for the period 1983-1992, 
again preceded by an important self-study process. 

The long-range planning has been integrated with short-term yearly programming and 
budgeting, and with assessment activities that include: 

-A locally designed ability testing system for the yearly admission of new students that 
has been used since 1976; in Mexico there is no nation-wide system of testing. Some 
large private higher education institutes, such as Monterrey's Technological Institute 
(ITESM) are using the Spanish tests developed by tiie Collie Board in Puerto Rico. 
Starting in June 1990 tiie UAA will be die first Mexican public HEI to do die same. 

-A locally designed &culty evaluation system, also from 1976, including evaluations 
from students, dq>artment heads, and deans. Witii small changes the system is still being 
used in 1990, and has therefore produced an almost IS-year-old series of data that allow for 
longitudinal comparisons. 

-Starting in 1977, a permanent performance indicators evaluation system, including tiie 
publication of a yearbook entitied "Evaluation," already in its 12tii edition. 

-An overall curriculum evaluation for all tiie professional fields of the institution in 
1979-1980. Since tiiat time a n^ular system of cjrriculum evaluation has been functioning, 
witii each professional field (32 in 1990) being evaluated every five years. 

-In 1979 and again in 1983 q>ecial assessments of die University's library system have 
been carried out. 

-From 1984 on, a dropout research covering-until 1990-13 of tite 32 professional 
fields, using a complex causal model (related witii tiiose developed by Spady, Unto, and 
otiiers). 

-In 1985, a large p rospective stiidy, tiiat tried to forecast tiie needs of professionally 
skilled manpower for different ectmonuc branches, and tiie future growtfi of social 
demand for higher education, to assess the need for a new HQ in the State of 
Aguascalientes before tiie year 2000. 

-From 1987, a program of alumni research. 




-In 1990, the Planning and Institutional Research Office of the University (Diicccion 
General de Planeacion y Desanollo) is already beginning to prqjare the third "Plan de 
DesaiTollo", which will cover the years 1992-2001. 

As a part of the piepaxadon for this new planning process, a new evaluation scheme 
is being developed, to pass from a more traditional resource-centered to a new outcomes and 
also process-focused approach, taking into account the different evaluation-related activities 
already mentioned. 

The concept of "quality*, and q^edfically of "higher education institution "quality* 
that underlies the new evaluation system that is being developed can be described as follows. 



If we concq)tualize the higher education system as a central body of processes 
(academic, as teaching, leaming, research and seivice, and managpnent and support) fed by 
some inputs and producing some outcomes, and situated in an environinait that provides the 
system with resources and at the sune time has some needs that can at least partially be 
satisfied by the outcomes of the system, then it is easy to define the basic two dimensions of 
quality: efficiency and efifectiveoess. 

-Efficiency is the relation that exists between the inputs and the outcomes of the system; 
it is a cost-benefit relationship; we can ^eak of the products of the system as che^ 
or expensive in terms of efficiency. 



Figure 1 



The Education System 



ENVIRONMENT 



RESOURCES 



NEEDS 
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-Effectiveness is the relation that easts between the outcomes of a system and the needs 
of its environment that are sappoxd to be satisfied at least partially by it In terms of 
effectiveness we can speak about the products of a system as useful or useless. 
Hie effectiveness dimension of quality is closely related with some other very 
important concq>ts: 

-The relevance of the objectives and curricula of the «wf!!rafimial system, in relation 
with the needs that the biological, ecological and social environment represents for the 
HEI. 

-The proportion of each cohort tiiat has access to a particular level of die 
educational system. 

-The attrition-retention rate, that is the proportion of each oabsrt tiiat not only has 
access to die level, but is able to successfully complete it 

-The goals attainment level, diat is the proportion of the student body tiiat achieves 
the learning diat is expected: 

-The equity dimension that is closely related with the access retention and level of 
achievement questions: Imowing that nature and nurture dififisences also produce 
differences in students* access, permanence and achievement The equity 
dimension is the characteristic of the system that reduces those differences. 



Rgure2 



THE EVALUATION COMPARISONS 



PROCESS EVALUATION 
TO OPEN THE BLACK BOX 




EFFICIENCY (Satisfaction or needs) 

(Use or Resources) 

(IMPLIES RELEVANCE 
& EQUITY) 
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Syn i aU these dimensions of quality, we can state that a quality education is 
one that: 

-has cially relevant educational puiposes and cumcula; 

-makes a substantial part of each age group reach those goals (by giving access to 
the particular educational level to the socially adequate proportion of each 
cohort; by retaining a very high proportion of them until the end of the level; 
by making them achieve the educational goals established by the curriculum; 
and by helping more those with more difficulties in order to achieve the 
former points;): 

-and does that in the most economic way. 

Another way of stating the same is to say that a quality education is one that is at the 
same time relevant, equitable, effective, and ef^dent. 

The importance of establishing precisely these differetu quali^ dimensions can be 
s^reciated if we consider tfiat very often only the efficiency dimension is considered. As it 
is much more difficult to assess the relationship between educational outcomes and social 
needs than to assess the relationship between inputs and outputs, very often only the latter is 
considered. 

As an "advert:sing-alienated consumer,* that is ready to buy anything without 
knowing its utility because it is cheaqp, many higlier education institutions are concerned only 
with cost-benefit analysis (student or graduate per capita expenditures), without caring about 
the value of the contributions their graduates are offering to the society in which they live 
and work. 

Taking this definition of educational quality, we present in the following pages a set 
of performance indicators for the evaluation of a HEI. Some of the indicators have been 
used in the UAA since 1976; others are being proposed for use in the self-study process that 
the UAA is preparing for its third Plan do Desarollo. 

The set of performance indicators includes some related specifically to one or another 
of the quality dime. '«t described: relevance, equity, effectiveness, and efficiency. 

As in many case& ' ^o obtain all the informaticm needed on some of the 

institutions' inputs and outcoi» ntion is also paid to process, and some process 
indicators are included in the set. e constraint ourselves to the teaching fimction, as 
research and service are less develop i in the UAA. 

In almost all cases the indicators can be calculated at the more global (or institutional) 

90 



BEST COPY AVWLftBLE 



level of aggregation, or at the more specific professional field (or career) level. In the same 
way, it is possible to calculate the indicators for one year or for some or many consecutive 
years, in order to know the dynamics of the processes concerning variation rates. 

Product Indicators 
Effectiveness indicators: 

Social demand j^proach: Proportion of the social demand satisfied 

% potential - social demand (global) 
% real - social demand (global) 
% potential - social demand (by field of 
studies) 

% real - social demand (by field of 
studies) 

proportion of male-female students admitted 
proportion of low-income students admitted 
proportion of national vs. international students admitted 
proportion of students applying for scholarships or loans 
social needs s^roach: 

diversity of professional fields (students by professional field) 

adequacy of professional fields to social needs 

- % enrollment by area to national goals 
* % of curricula developed in relation to 

social needs 

- % of graduates working in their professional field 

(after six months or more) 
> % of graduates widi an outstanding 
professional record 

Efficiency indicators: 

Total student per capita expenditures (yearly per student cost; global and by professional 
field) 

Total student per capita expenditures (five-year periods per 

graduate cost; global and by professional field) 
Yearly attrition rate (global and by professional field) 
Generationai or cohort attrition rate (global and by professional field) 
Professor/student ratio (total) 
Professor/student ratio (full-time) 
Professor/student ratio (full-time equivalent) 
Mean group size 
Qerical staff/professor ratio 
Clerical staff/students ratio 
Square meters of area per student: - total 

- buildings 
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- g ds 
- 1: 7 spiot 
library vc nes per student 

% c issroom utilization (100% * 70 hours per week) 

Teaching and Timing Process Indicators 

Cuniculum definition and evahiation: 

-Existence of general cucricular objectives explicitly related to environment needs. 
-Existence of a cuniculum periodical evaluation system. 
'% of course programs or syllabi systematically developed and related to general 
curriculum (^jectives. 

Student activities: 

weekly course load (mean global or by professional field) 
weeidy hours of independent study (mean global or by professional field) 
hbnry utilization: mean of weekly hours (global or by professional field) 
grade point average 

% of students £uling to pass one or more courses 
student attrition rate 

student participation in special learning activities 

(remedial activities, counseling, social service, etc.) 



Faculty activities: 

Existence of presence control for fitculty 
% of classes in^nrted vs. programmed 

Existence of dqnrtmental control over the teaching and learning process 
% of dqnrtment-controlled achievement testing 
Existence of fiKulty evaluation systems 
Mean of student ratings of teaching 
Existence of facalty development programs 

% of fiunilty having taken at least 6 credits of fiaculty development activities per year 
% of the fiunilty direct teaching load 

" teaching-rdated load 

" research load 

" service load 

" manar'*""^t load 

Inputs-Resoyrrn< Tnriiratoi 
% fiill-time hadty vs. 
% tenured facxlty vs. tou 
% Ph.D. or equivalent £ku total 
% faculty with at least a fiill of graduate studies vs. total 
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% £iculty members by status and category 
total student enicllment; rate of increase; 

enrollment by professional field; rate of increase; 
% of new students s^ve or below different levels of adiievement, according to the 

admission test 

% of budget by function (teaching, research, service, management and support) 
% of budget by concept (salaries, general expenditures, buildings and equipment) 
% of income coming from federal or state fimding or from tuition rates or other own 
resources) 

Conclusion 

We stated before that it is easier to evaluate efliciency than effectiveness. 
Now we have to add that this is not due to due technical side of the question, but to the 
philosophical or ideological one. ' If we submit the concq>t of social needs to scrutiny, we 
can see that a signifies a relationship: the distance that sq>arates the situation as it 
is in this moment, and the ideal situation, the desirable would-be. If it is not difficult to 
obtain a consensus about what something really is, it is much harder to concur about what we 
would like that thing to be. 

We have to start with a broad jncture of the kind of ideal society we would like to 
have in order to establish the QfifidS of our present society, and decide, tfien, which will be 
the fundamental purposes of our HEI in order to contribute to the satisfaction of those needs, 
and finally to design the maos (curricula, contents, method, teachers, etc.) that will 
eventually allow us to achieve the purposes and to satisfy the needs. Once the social needs 
and correlative institutional purposes have established the technical side of the question, the 
means-and that of the evaluation indicators-can be met. 

Of course a total consensus about needs and purposes is not feasible nor indeed 
desirable, but a minimum broad philosophical woridng consensus is the sine-qua-non 
condition for advancing both the construction and evaluation of a higher education institution. 
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The Research-oriented Dynamism of Education 
Jose Stmbbe 
University of Ghent 
Belgium 

Introduction 

Universities perform two fimctions: teaching and research. Indeed, the key 
principles of a university are accentuated in its goals: widening the frontier of knowledge and 
transmitting this knowledge (even in the undeigraduate programs). Both goals have their 
appropriate activities, although sometimes arbitrary, which distinguish them from each other. 
Considering the scarci^ of literature in this area, the starting point for discussion of the need 
for this connection was *he Robbins Report, which held the essence of a university to be both 
research and teaching. Although a wider range of literature presents a less clear statement, 
according to the Reynolds Report, "The interrelationship between teaching and research is an 
essential means of keeping curricula up-to-date through the involvement of academic staff in 
original work and inquiry." 

The University as a Managerial Organization 

In general terms, organizations can be considered as a series of procedures, functions, 
and structures: the university organization realizes both functions through two fields, one 
related to purposes, the other to resource allocation. Both have to take into account the 
principles of effectiveness and efficiency CFigure 1). Quality has an optimal situation when 
all fields overlap each other. This situation can be reached when w strategic policy is built 
up with 'Elements such as puiposes, goals, and planning and, further on, with a strategic 
zpppioach. From this introductory point of view, quality ineasurements have a certain 
interdependency: several items influence the whole approach on the organizational level of a 
higher education institution as wtU as on the level of postsecondaiy education in general. 

Program Organization 

From the education side, the program organization process is based on four levels: 
curricula, teachers, infrastructure, studbits. Each of these cat^ories has its own features, 
strategies, particulars, and consequently its own measurement of quality. In this way each 
class forms a quality-circle in which measurement can be staled and referred to a certain 
quality-reference point. Attempting to coordinate the whole measurement process brings up 
the problem of ranldng between the quality-circles although the quality of tht entire process 
prevails over each quality-circle. 




Purposes: Effectiveness 
Tension-Held 




Tensim-Field 
Resource Allocation: Efiidency 

Figure 1: The managerial situation of research 
(and education) in a university 
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Program/Organization Chart 




ThCTC again, interdependency of measurements influences the whole process, and 
should be divided into two areas depending on items relating to effectiveness or to efficiency. 

Quality Measurements 

Out of the principle of effectiveness arises the question about degree of realization. 
While the principle of efficiency brings up the question of the most rational use of resources, 
it is connected with management and quantitative measures. Effectiveness, und^tood only 
too well in private companies as a quantitative measurement, is not useful in the context of 
the university, because neither goals nor results are quantifiable. 

Again, the support and improvement of effectiveness for the most part will be found 
in the content of educat. - tjrch-oriented policy-measures and in the control of the 

processes involved. Critc. iveness are organizational productiviQr, flexibility in 

adjusting to both external ana envux)nmental change, and absence of strain or tension among 
organizational subgroups. 

In this way these structural subgroups must have the same degree of importance and 
influence or, at least in the subgroup quality-circles, need a well-stated ranking. 

In Figure 2, the top level of this ranking represents curricula and teaching, while the 
second level represents infrastructure. However, in reality sometimes doe inverse is true. 

Fdncarinn and Research Infl uences as Characteristic In Deoth for the Measurement of Quality 

The above mentioned criteria for effectiveness not only refer to teaching dynamism 
(transfer of knowledge, methodology, intopretation, results and conclusions are the 
educational foundations) but also to research-oriented dynamism. The former is certainly 
situated in the teaching-circle (see Figure 2) while the latter has different roots. 

Indeed, the first steo for the orogram organization-making a choice in curricula-is 
certainly based on, or i ^ by, the state of the art of science and research 

activities in the program 4»_ . al and the chosen program in particular. Well- 

structured subdivisions of research disciplines and worked-out research areas are preferred 
for graduate and/or postgraduate studies. 

When carrying out in-dq)th analysis of the essential characteristics of a curriculum 
and its relationrhips with other circles (see Figure 2), several statements come to the fore: 

—The learning outcomes of each curriculum counts of difftnnt Ir/els: 
-knowledge (facts, concq)ts, gen&,<dizations) 
-techniques (pioccssc<:, skills, abilities: cognitive and psychu-motor) 

—The sources from which a curriculum is selected is the available culture 
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which refers directly to the state of the art of that research area and/or 
discipline. 

-There are "necessary" criteria involved in the selection of curricula such as 
teachability and leamability, whereas other criteria may refer to ideology 
(society, political, social, economic or moral values), utility, and the relative 
significance within the structure of intellectual disciplines. 



-Structure, which refers to the order of items mandatory for instruction, is another 
essential characteristic of curricula. 

—Curriculum guides instruction. 

Consequently, the effectiveness of education and program (together with instruction 
and teaching) is represented by the extent to which actual outcomes correspond with intended 
outcomes (congruency). This refers to the different quality-circles (see Figure 2) and at least 
to those of curricula, teaching, and students. 

R eality 

Keeping these statements in mind and relating them to the improvement of quality, 
some universities in Europe are creating subject groups concerned with the organization, 
coorduiation, and integration of scientific research and education. These are important tools 
in developing proposals and working out the content of educational and research-oriented 
policy measures. 

They also work as a peer-review system. The different subjects of a curriculum are 
reviewed every five years by testing. (In reality the subjects are all parts of curriculum). 

The Belgian universities and authorities are concerned with a permanent evaluation of 
tiie "education-h^pening," in order to guarantee tiie scientific level and tiie social relevance 
of each degree. In this way, tiie Flemish Decree 1992 provides a five-year annual evaluation 
of university activities. 

Conclusion 

The measurement of quality in postsecondary education is very wide ranging, and 
research-oriented dynamism of education is a significant charter. Therefore, subject-groups 
have an important descriptive (though not exact) and judgmental task. 



99 

lu Z 

ERIC 



Bibliography 

Bellack, A. and H. KUebaid. Cumculum and Evaluation . 1977. 

Brown, 2*1. K. "Developing and implementing a process for Ae review of nonacademic 
units." Research in T Tigher Education 30(1). 

Frissen, P., Van Hoewijk, and J. Van Hout. De universitcit: een adequate 
onderwiisorganisarift? 1986. 



Magennis, S., B. White, and R. Woodward. "Teaching and the University Curriculum: 
Influences from Research." Contributed Paper to the 13th International EAIR- 
Forum, 1991. 

Pfeffer, J. "Usefulness of the Concept" in New Perspectives on Qrpani zational Effecrivmff^s^; 

• "Educational Foundations for Career Hexibility," in Th e Research System in 

Transition . 

• "Changes in Agriculture and Environment: A Challenge for Higher Education" 

(European Conference on Higher Education in Agriculture). 

Reynolds, P. A. Academic Standards in TTniverCTrifts, London, 1986. 

Robbins, A. Higher Educntinn Report of the Committee, London, 1963. 



100 



Q tiEST copy AVAILABLE 

ERIC J-Uo 



Explaining Differences in Drop-out Between Faculties 
in Dutch University Educrion 



Riekele Bijleveld 
Center for Higher Education Policy Studies 
University of Twente, The Netherlands 

Introduction 

Statistical data on drop-out rates reveal strong differences between faculties. Almost 
automatically the question can be raised: how can these differences be explained. For this 
specific project the research question would be: How can differences in drop-out rates 
between Dutch university-programs be explained? A more or less traditional research design 
containing a theory, hypodieses, operationalization of coacepts, choice of techniques of 
analysis and empirical testing, is used. The project is still in the stage of theory-development 
of which a brief impression will be given below. 

In general, drop-out can be related to student characteristics (background and 
intellectual cs^ities) and educational factors (teacher performance, content, level, 
organization, planning, counseling, etc.). Although student chazacteristics are important we 
decided, for various reasons, to focus on features of the teaching process. First, not much 
research has been done in the Netherlands involving more than one fiodty. Second, a great 
deal of the Dutch research in this area has been devoted to student characteristics whereas 
less attention has been paid to institutional and teaching factors. 

A few concepts imparted in our model are derived from a teaching approach practiced 
at the University of Limburg. TTie selection of these concq)ts is justified by this university's 
low attrition rate compared to that of other universities. The University of Limburg is 
characterized by its high commitment to the effectiveness of teaching and the subsequent 
substantial efforts to ititvM and improve the teaching process, an essential difference widi die 
other Dutch universities. Though these universities tiy to improve the feedback mechanisms 
regarding student performance, including die counselling of students, the leading principle 
still seems to be that every student is responsible for his swn performance. This system 
relies heavily on the individual student's motivation and self discipline, a conclusion 
supported by a few oUier observations. 
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(^han^cteristics o. Teaching Procfi5;5; at the Traditional Dutoh University 



First, the tniditional Dutch university program consists of many different courses 
(subjects) with Uie latter repi-esenting the spedaiizations in the discipline. During a term 
(trimester or semester) three or more courses are given simultaneously by means of general 
lectures and seminars. A term ends with a poiod of exams for each course. Participation in 
an exam is voluntary* A considerable number of students only participate in a selection of 
the offered exams aiid wait for a second or even third chance for the same course. This 
^strategy maximizes the net study-time for a certain exam at the end of a term but leads 
inevitably to a duration of studies that exceeds the nominal program length, an outcome that 
demotivates the student. 

The performance assessment procedures also contribute to a certain drop-out rate. 
Normally examinations are taken in written form. Due to the decentralized organization of 
student performance tests, conection*standards will differ between the exams and the 
correctors. This situation results in the use of different criteria for failing or passing an 
exam. In fact criteria depend on the performance level of the group subjected to judgment 
A student in a relatively weak group has more chance to pass a given exam than a person of 
the same level in a better performing group. This psychological mechanism stems from the 
hct that a judge tries to avoid extreme outcomes (very high or very low percentages of 
failing students), and explains why attrition rates of different courses are inclined to be stable 
in time. 

To a large degree, responsibility for the content of the courses has been assigned to 
the staff members as are the criteria used to establish the workload. This situation results in 
the establishment of courses which formally are considered to have the same workload, but 
in practice sometimes show great variation. It seems reasonable to assume that heavy 
workloads do not stimulate retention. 

Finally, it is important to note that teaching activities do not hold the same prestige 
and status as scientific research; teaching experience plays a secondary role. 

Characteristics of the Teaching Proces s at the University of Limburg 

The University of Limburg practices a different educational teaching philosophy, as 
briefly described below. 

Courses are planned as a seqt :^nce of six-week modules. During the six-week period, 
a group of students joindy analyzes a case, a task that requires multi- and interdisciplinary 
approaches. The group meets two times a week and discusses individual contributions to die 
project. The group is tutored by a member of the aca(iemic staff. In a project-meeting new 
tasks are discussed and assigned to the group members. At the end of each period a test is 
given to measure what has been learned to Uiat point. This test is not meant to evaluate 
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performance. For that purpose a completely new instrument was developed. 



A multiple-choice test with thousands of items has been constructed that exanunes 
knowledge of an entire discipline. Every semester a sample of items is drawn from aU these 
items. This soK^alled performance-progression test has to be taken by all enrolled students, 
independem of the stage of their studies. TTus system enables the observer to compare tiie 
performance of an individual student with his former results and also provides a compmson 
between students in a study stage or different stages. Due to the feet that all students tove to 
take the same test, a high leveJ of objectivity, vaUdity, and rdiabiUty has been achievai. 
Students whose performance is too for bdow the mean of their reference group are offered a 
compensation program to remedy their deficiencies. It is important to note ttat an important 
risk of the traditional system has been overcome, namely the dependence of the results on the 
specific criteria used for a certain examination. 

However other advantages of the system are also obvious. First, students actively 
participate, not only by the regular assignment of individual tasks but also by the discussions 
The student cannot esaxpe to other courses. Group feedback forces him to do his job and not 
to wait until an exam takes place. Motivation is also improved by the mter- and 
multidisciplinary ^roach. Once a concrete and recognizable problem has been identified aU 
relevant knowledge must be collected and internalized. A student learns that icahty should 
not be experienced as a system of relatively isolated sectors but as an entity with different 
aspects. 

The extireme specialization widiin the traditional educational system suggests that 
subjects have to be stiidied separately. GeneraUy this approach has a demotivating mfluence 
on a stoident. Lack of coherence between subjects does cause one to wonder why a certam 
subject has to be Stiidied. TTicory without references to reality is often exper^ced as 
useless abstraction. TTie project-centered approach should overcome th^ difficulties. In 
scheme 1 we summarize a few characteristic differences between the traditional and the 
Limburgian system. 

Does this unique approach have a negative effect on educational level? Research on 
this question indicates that graduates of the University of Limburg arc not mfenor to 
grad 'ites from otfier Dutch universities. 

Hiwever one should take care not to idealize the Umburgian system prematiirdy. 
Until now it only has proved its value for the medical discipline, a field in the Netherlands m 
which stiidents traditionally perform weU. In addition, the University of Limburg has a 
relatively smaU stiident population. FinaUy, the teaching task of the academic staff is 
assumed to be heavier compared than tiiat of tiie traditional umversities. 

Nevertheless it can be concluded that it is possible to reform the educational process 
without a loss of quality and without severe selections mechanisms, leading to a decrease m 
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A Provisiftnal Scheme for thP Taplanation of Differences in nr o p-nnt iv-twff f n Fn nilrir^ in 

Dutch University Krinrati^^n 

The observed differences between the teaching systems discussed above have been 
used to develop a theoretical scheme that may explain the differences in attrition between 
faculties in general. A provisional scheme (Figure 1) has been developed that relates 
teaching and factors such as focus on problem-solving, interdisciplinary approach, stimulatii 
of active participation of students, frequent student-staff interactions, intensive student 
counseling, sequential programming of course-load, an even distribution of the work-load 
working in groups, entrance-selection, level and performance-measurement, to the drop-out 
rate of a faculty. The different theoretical constructs have to be operationalized. Data 
needed for the operationalization of the concq)ts will be derived from written sources and b 
means of questionnaires. Not all the faculties in the Netherlands can be included in the 
project. They will cventuaUy be matched by selected features like size and drop-out rate 
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Appendix A 



Participants at tlie September Meeting 



Trudy Binta 

The Univenity of Tennessee 
1819 Andy Hoi: Towen 
KnoxviUe,!?? 37996 

Phone: (615)974-2350 
Fax: (615)974-2712 

MAiianne Bauer 

Natiooil Boend of Universities 

A Colleges 
Box 45501 
10403 Stockholm 
Sweden 

John Brennan 
CNAA 

344 54 Gny S Inn Road 
London WCK 8BP 
United Kingdom 
Phone: 071-278-4411 
Fax: 071-833-1012 

Elaine El-iChawu 

American Council on Education (ACE) 
One Dupont Circle, NW Smu^ - >} 
Washington, DC 20036 
U.S.A. 

Phone: (202)939-9310 
Fax: (202) 833-4760 

Edgar Fiickmann 

Hocfaschul-Informations-System 
BP 2920 

3000 Hannover 
Genaany 

Fax: 49-511-1220-250 

, Robert Froh 
Syracuse Univenity 
Testing Senrices 
209 Archbold Gym 
Syracuse, NY 13244 
U.S.A. 

Phone: (315)443-2295 
Fax: (315)443-1524 



Leo Goedegebuure 
Univernty of Tweate 
CH.E.RS. 
P-0. Box 217 
7500 AE Enachede 
Hie Netherlands 
Phone: 31-53-893263 
Fax: 31-53-340-392 
E-Mail: 

GOEDEBUUR ® UTWENTE.NL 

KariHyppoenen 
Universi^ of Turku 
20500 Turku 
Finland 

Derek Jamiesoo 

Univeni^ of Guelph 4th Floor 

Gudph. Ootirio 

NIG2W1 

Hwoe: (519) 824-4120 (X3456) 

Fax: (519)767-1693 
Bitnet: ANPDEREK ® UOQUELPH 

PMer Mufwa 

Univeni^ of Tweate, CHEFS 
P.O. Box 217 
7500 AE EoKfaede 
The Netherlandi 
Phone: 31-53-893263 
Fk: 31-53-340-392 
Bitnet: MAAS ® UTWENTE.NL 

Aims McGuinaeet 

Educatioa Conmiitsiaa of the Stitet 

707 17di Street, Suite 2700 

Denver, CO 80202 

U.S.A. 

Pbooe: (303)299-3614 
Fix: (303)296-8332 
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Michael Nugent Peter Weusthof 

Center for the Study of Higher Education University of Twente, CHEPS 

The Pennsylvania State University P,0* Box 217 

403 South AUcn Street, Suite 115 7500 AE Enichede 

University Park, PA 16801-5252 Tlie Netherlands 

U.S.A. Phone: 31-53-893263 
Phone: (814)863-0854 Fax: 31-53-340392 

Fax: (814)865-3638 
Bitnet: MAN114®PSUVM 

James Ratcliff 

Center for the Study of Higher Education 
The Pennsylvania State University 
403 South Allen Street, Suite 104 
University Park, PA 16801-5252 
U.S.A. 

Phone: (814)865-6347 
Fax: (814)865-3638 
Bitnet: JLR7®PSUVM 

Allan Sensicle 

Hong Kong Council for Acadademic 

Accreditation 
14/F* Ruttoojee House 
11 Duddell Street, Central 
Hong Kong 

Phone: 801-7480 
Fax: 845-9910 
E-mail: HKCAA in care of CPHKVX. 



Frank Schmidtlein 
University of Maryland 
Room 4114 CSS Building 
College Park, MD 20742 
U-S*A* 

Jose Strubbe 

University Gent 

St Pietersnieirwstraat 25 

9000 Gent 

Belgium 

Phone: 091-64-30-37 
Fax: 091-64-35-97 



Don Westerfaeijden 
University of Twente, CHEPS 
P.O. Box 217 
7500 AE Enachede 
Hie Netherlands 
Phone: 31-53-893263 
Fax: 31-53-340392 
E-n»il: WESTERH © UTWENTE.NL 
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CUfiford Adds 

Divuioa of Hig.. . Educatioii 

Office of Research 

U.S. Deputmeot of Educatioa 

555 New Jeney Ave. 

WaihiAgton, DC 202208 

U.S.A. 

Phone: (202) 357-6244 
Fax: (202)219-2030 

Reikele Bijleveld 
Univenity of Twente, CHEPS 
P.O. Box 217 
7500 AE Emchede 
The NetberUndi 
Phone: 31-53-893263 
Fax: 31-53-340392 

Sal Comllo 

OERI Oflice of Reaearch 

NCRES 

Room408F 

555 New Jmey Ave, NW 
Waihington, DC 20208 
U.S.A. 

Phone: (202) 219-1913 
Fax: ( 202) 219-1676 

]eifny Gilmoie 

OERI Office of Reteaich 

Higher Educadcn Sl Adult Leaniing 

Room 6158 Capitol Place 

555 New Jeney Ave. 

Waahingtoo. DC 20208-5647 

U.S.A. 

Phone: (202)219-2254 
Fax: (202)219-2030 
Bitnet: Cno^NraCU 

Steven Hunt 

OERI Office of Reaeaich 
555 New Jeney Avenue, NW 
Waahington, DC 20208-5647 
U.S.A. 

Fhooe: a02) 219-2253 
Fax: (202)219-2030 



Additional Fartidpaots 

Felipe Martinez Rizo 
Univ. Autonoma de Aguascalieates 
San Cosme N 108 
Aguaccalientei 20010 Mexico 
Phone: 49-14-20-29 
Fax: 49-14-55-91 

Michael Nettlei 

Vice Pneident of Aaaecament 

The Uaivendty of Teonesaee 

817 Andy Holt Towen 

&ioxvilie,TN 37996-0185 

U.S.A. 

Phone: (615) 974-4815 
Fax: (615) 974-3213 
BriNET: PA1234580UTCVM1 

Hana Pecfaar 

Miniatiy of Education and Science 
Banlrgiaaw 1 
A 1010 

Vienna, Auatria 
Gaiy Pike 

The Caater for Aaaeaament Reaeaich 

& Developnieat 
UniveniQr of Tenneaaee 
1819 Andy Holt Avenue 
Khoxville, IN 37996^350 
U.S.A. 

Phone: (615)974-2350 
Fax: (615)974-2712 
Bitnet: PA108968 (|( UTKVMl 
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Stefuie Schwiiz 

Center for the Study of ffigher Educatioa 
The Peansylvani* State University 
403 South Allen Street. Suite US 
University P«k, PA 16801-5252 
U.S.A. 

Phone: (814)863-0854 
Fax: (814)865-3638 



David Woodhouse 
Executive Director (Acting) 
Hong Kong Council for Ac«lemic Accreditation 
14/F. RuttoojeeHouae 
11 Duddell Street, Central 
Hong Kong 
Fbooe: 801-7480 
Fax: 845-9910 
E-mail: HKCAA in care of CPHKVX. 



Vincent Tinto 

Cultund Foundations of Education 



Baihara Wright 

American Association for Higher Education 
One Dupoot Circle, Suite 780 



Syracuse University 
259 Huntington Hall 
Syracuse, NY 13244-9144 
U.S.A. 

Phone: (315)443-9081 
Fax: (315) 443-5732 



Waahington, DC 20036 
Fhooe: (202) 293-6440 
Fax: (202)293*0073 



BITNET: VTINTO®SUVM 

Michel van Buchem 
University of Twente, CHEPS 
P.O. Box 217 
7500 AE Fnsrhfdft 
The Netheriands 
Phone: 31-53-893263 
Fax: 31-53-340392 

Franz Van Der Meer 
Kotteidam Inat for Higher Education 
^'iKhussenstrsat 125, 3015 £J 
Rotterdam. The Nedieilanda 
Phone: 010-4361233 
Fax: 010-4367033 

Maryellea Weimer 

Center for the Study of Hi^ Educati<m 
The Pennsylvania State University 
403 South Allen Street, Suite 104 
University Paik, PA 16801-5252 
U.S.A. 

Fhooe: (814)865-8366 
Fax: (814)865-3638 
BitneC GRG^PSUVM 
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Appendix B 

1 JNAL CENTER ON POSTSECONDARY TEACHING, 
LEARNING AND ASSESSMENT 

over iO^^rlvemto. b«S «ith thdx ftehmo. year »d conanumg taough 
their junior year. Hie two major "-als of the study are: 

■ to Ktimale ho - • and nonacademic experiences influence student 
SJITS Etudes towarf leanung. their cognitive devrel^OKnt. and «tar 
SScetoiS (Academic expeiiencesinchide the quality of ttactang 
S^S^m^». the lev^F^t involvement in academe work. 
^ SnXm of coursework tatoi. Nbnacademic experiences inctode 

^yities. relationships wiU, faculty ».d peers, and work and 

family responsibilities*) 

■ to determine the extent to which these academic experiences d^ among 
^^^Sents attend^g different kinds of colleges and ^^^'^l 
vanous ™wcn» *^ «\ minority and non-mmonty stiuknts 
^^r^^S^ ^TX^sWdents in ways uriike yo».ger 

^^T^ nr^S^ do the diffiwnt kinds of institutions-two- 

commuter a»l residcnti^-afl^ the 

college experiences of students?) 

R««irf, Prolan o» to Cj«to^ 

Two cpiestions define the ptoblonaflare^ «^ 

constitutes genetal 1. . «™"«rSt^^^^of the itepaduale 
Kesearch-b««l tao». - -bout *«,^^^i^^ve abiUties «ul 4e values and 
cuniculum atody enst. - '"^fJ^ Ato » »« understanding of how 

„<^,«^bystude.^to_ ^^i^ectivec«mcula.T^P«3ec.s«ill 
that knowledge base sdHUB fllS Maiflt aifi ESWafi EaBBIlS 

SlS^^eS *^To^r=l"Si,SL^^SS^". tenns of how 
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it is defined and identified. 



nMsarch Proeram on Faculty and Instruction 

"""^ll^sfrclassroom Lmains the focal point of the learning ^^^^^^yf 
collegeS student learning must examine the dassnx,m expeneno^ ^^^^.u^^^L 
smd^ Of the five pioiecte to be undertaken in this program, two focus on the fecu^ty and 

their roles and how they learn the norms of the P^^^^^V"^ 

specifically concerning teaching. The Eajailtt EffifilC P^ject will develop » «>™f«?«^^^^® . 
Sption ^ A^i's postsLndary teachers in terms of demographic charactenstics ^d 
trSties m proj^ on Enteinfi includes s^eral ^^^^^^J'^^ 

and use of various kSids of information by faculty to assess and ^«?P«>ve mst^d^on ^e 

InsttuciiQiial MfiihQds and Mtoto SludsnJs Project wiU use ^^K^^^Jf^ 

SSTiSfySstructional methods compatible with the ways mmonty students learn m 



community colleges. 



Rfiseareh Ptozram on Out-of-Clas^ Experiences , , . 

^^portST^ the curriculum and classroom are in shapmg student l^armng am^ 
evidence tadicates that the instructor and formal instructional cxpen«iccs are not all d«4 
^ea ^^T^g. As much as 85 percent of a typic^ undergraduate's waking Jioun are 
^ouSde formal Ltructional settings. THese out-of-class «q)en«iccs .««^y 

classroom learning, produce new learning, or even reduce learmng by fi^ncttomng 
^^wS^T^uS nLion of the institution. Because of die important mflu^ces of 

«S«ic^ on learning and retention, this research program will «Plorc: (1) how 
S,!dLS^^aSve and Sivolved participants in an academic commumty, and (2) how 
sS^S^'^-^fX^«U paSarty dieir interpersonal interactions (whetha- on- or 
S^^pust^cinforce^augment-^ weaken-curricular and classroom learmng and 
achievement of broader general education goals. 

Research Frozram on Organizational Structure and Policies ^ 
Researdi«^^ Ls toS^ administrative approaches to improvmg undergraduate 
teachinrST^assessment: Researchers here will pay particular attenuon to how 
SS^aSi^on have created and/or can create an effective learmng climate for 
^^L^d^ups such as African Ameiicansmiack Amencans Native 
rS^Sdca Hispanics/Latino Americans, women (in tnditionaUy 

md«^sented fields like science), and the disabled. THey will mv«mgalc: (1) die 

du^tcristics that may either help or hinder die P^shjor "movatoon in 
^S^d l<S^the information and data necessary to make effective deaaons 
"^^^^IS^^Z^ (3) leadership activities necessary to create a chmate 
for change. 

and ImproTement* 
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